---
url: https://jer-k.github.io/effect-prophet/examples/tuning.md
description: >-
  Choose settings with comparePerformance and searchModels, compare against
  baselines and check on a final holdout.
---

# Choosing settings

The defaults work well for a lot of data, but you can often do better by trying a few settings
and keeping the one that forecasts best.

::: tip Benchmarked against Python Prophet
Settings search and the final holdout check are benchmarked on all three trend types. See
the [cross-validation and choosing settings benchmarks](/benchmarks/cross-validation).
:::

## Comparing settings

`searchModels` runs [cross-validation](./cross-validation) for each set of options you give it,
on exactly the same rounds, and picks the one with the lowest error.

```ts
import { Effect } from "effect";
import { prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const history = Array.from({ length: 120 }, (_, day) => ({
  timestamp: date(day),
  value: (day < 70 ? 50 + day * 0.2 : 64 + (day - 70) * 1.2) + 3 * Math.sin(day * 12.9898),
}));

const result = await Effect.runPromise(
  searchModels(history, {
    // Each candidate is a set of fit options you want to try.
    candidates: [
      { id: "stiff-trend", options: { map: { changepointPriorScale: 0.001 } } },
      { id: "default-trend", options: { map: { changepointPriorScale: 0.05 } } },
      { id: "flexible-trend", options: { map: { changepointPriorScale: 0.5 } } },
    ],
    plan: {
      horizonMs: 7 * dayMs,
      cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
    },
    objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
    // "record" keeps going if one candidate fails; "fail-fast" stops at the first failure.
    failurePolicy: "record",
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

for (const candidate of result.candidates) {
  if (candidate.kind === "success") {
    console.log(candidate.id, candidate.score.toFixed(3));
  } else {
    console.log(candidate.id, "failed:", candidate.failure.tag);
  }
}

console.log("best:", result.selected.id);

```

```txt [Output]
stiff-trend 12.556
default-trend 3.459
flexible-trend 2.907
best: flexible-trend
```

This data has a sudden change in growth, so the most flexible trend wins. Good settings to try
first:

* `changepointPriorScale`: try `0.01`, `0.05`, `0.1`, `0.5`
* `seasonalityPriorScale`: try `0.01`, `0.1`, `1`, `10`
* `holidaysPriorScale`: try `0.01`, `0.1`, `1`, `10`, if you have events, country holidays or
  regressors
* `seasonalityMode`: `"additive"` or `"multiplicative"`

`seasonalityPriorScale` and `holidaysPriorScale` set the prior for every seasonality, or every
event, country holiday and regressor, that doesn't set its own `priorScale`. Lower values keep
those effects small. As in Python Prophet, regressors follow `holidaysPriorScale`, so give each
regressor its own `priorScale` if you want to tune holidays alone.

You can compare up to 32 candidates in one search (`maximumSearchCandidates`).

To pick the winner by a score of your own, pass a
[custom metric](./cross-validation#your-own-metrics) as `objective.metric`. The final holdout
check below needs a built-in metric.

## Trying every combination

Rather than writing out each candidate, give `makeParameterGrid` the options every candidate
shares and the values to try for each setting. It returns one candidate per combination, ready
for `searchModels`:

```ts
import { Effect } from "effect";
import { makeParameterGrid, prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

// Growth speeds up on day 70, and a weekly pattern grows with the trend.
const history = Array.from({ length: 120 }, (_, day) => {
  const trend = day < 70 ? 50 + day * 0.2 : 64 + (day - 70) * 1.2;

  return {
    timestamp: date(day),
    value: trend * (1 + 0.1 * Math.sin((2 * Math.PI * day) / 7)) + Math.sin(day * 12.9898),
  };
});

const result = await Effect.runPromise(
  Effect.gen(function* () {
    // Every combination of the values below: 3 × 2 = 6 candidates.
    const candidates = yield* makeParameterGrid({
      base: { builtInSeasonalities: { daily: "off", yearly: "off" } },
      axes: {
        changepointPriorScale: [0.01, 0.1, 0.5],
        seasonalityMode: ["additive", "multiplicative"],
      },
    });

    return yield* searchModels(history, {
      candidates,
      plan: {
        horizonMs: 7 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
      },
      objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
      failurePolicy: "record",
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

for (const candidate of result.candidates) {
  if (candidate.kind === "success") {
    const { map, seasonalityMode } = candidate.options;

    console.log(
      candidate.id,
      map?.changepointPriorScale,
      seasonalityMode,
      candidate.score.toFixed(3),
    );
  }
}

console.log("best:", result.selected.id);

```

```txt [Output]
grid-0-0 0.01 additive 7.236
grid-0-1 0.01 multiplicative 5.006
grid-1-0 0.1 additive 2.946
grid-1-1 0.1 multiplicative 1.822
grid-2-0 0.5 additive 2.773
grid-2-1 0.5 multiplicative 1.685
best: grid-2-1
```

These settings can be varied:

| Setting                 | Replaces                    | Example values                                                   |
| ----------------------- | --------------------------- | ---------------------------------------------------------------- |
| `changepointPriorScale` | `map.changepointPriorScale` | `[0.01, 0.1, 0.5]`                                               |
| `changepoints`          | `map.changepoints`          | `[{ mode: "auto", range: 0.8 }, { mode: "auto", range: 0.95 }]`  |
| `seasonalityPriorScale` | `seasonalityPriorScale`     | `[0.01, 0.1, 1, 10]`                                             |
| `holidaysPriorScale`    | `holidaysPriorScale`        | `[0.01, 0.1, 1, 10]`                                             |
| `seasonalityMode`       | `seasonalityMode`           | `["additive", "multiplicative"]`                                 |
| `holidaysMode`          | `holidaysMode`              | `["additive", "multiplicative"]`                                 |
| `builtInSeasonalities`  | `builtInSeasonalities`      | `[{ yearly: { mode: "on", priorScale: 1 } }, {}]`                |
| `seasonalities`         | `seasonalities`             | `[[], [{ name: "monthly", periodDays: 30.5, fourierOrder: 5 }]]` |
| `scaling`               | `scaling`                   | `["absmax", "minmax"]`                                           |

Each value replaces the base's setting outright. A list or object, such as a set of seasonalities,
counts as one value; it isn't combined any further. To try one seasonality's own prior scale
instead of `seasonalityPriorScale`, give each version of the whole `seasonalities` list as a value.

Candidates come out in the order of the table, with the first setting changing slowest, whatever
order you write the axes in. Each ID is `grid` followed by the position of its value in each
setting you gave: `grid-2-1` above is the third `changepointPriorScale` and the second
`seasonalityMode`. With no axes, the one candidate is the base, with the ID `grid`.

`makeParameterGrid` checks the whole grid before you start a search, and fails with a
`SearchOptionsError` when:

* the grid has more than 32 combinations (`reason` is `limit-exceeded`, checked before any
  value is read),
* a setting has the same value twice (`duplicate-choice`), or has no values or isn't in the
  table (`invalid-shape`),
* the base options are invalid (`invalid-base-options`; the path starts with `base`),
* a combination is invalid (`invalid-candidate-options`, with the combination's position in
  `candidateIndex`). When the problem is in one value, the path points at it, such as
  `["axes", "changepointPriorScale", 1]`. When it comes from how values combine, such as a
  seasonality named like a base regressor, the path points at the candidate:
  `["candidates", 3, "options", ...]`.

For invalid base or combination options, `cause` holds the original `OptionsError`.

It fits nothing and needs no fitting backend.

## Comparing against simple rules

Before trusting a model, check that it beats the obvious guesses:

* **Last observation:** tomorrow will be the same as today.
* **Training mean:** tomorrow will be the average of everything so far.
* **Seasonal naive:** tomorrow will be the same as this day last week.

```ts
import { Effect } from "effect";
import {
  comparePerformance,
  crossValidate,
  crossValidateBaseline,
  prophetFittingBackendLayer,
} from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const history = Array.from({ length: 120 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 3 * Math.sin(day * 12.9898),
}));

const comparison = await Effect.runPromise(
  Effect.gen(function* () {
    const model = yield* crossValidate(
      history,
      {},
      {
        horizonMs: 7 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
      },
    );

    // Simple rules of thumb, checked on exactly the same days as the model.
    const lastValue = yield* crossValidateBaseline(history, model.plan, {
      kind: "last-observation",
    });

    const sameDayLastWeek = yield* crossValidateBaseline(history, model.plan, {
      kind: "seasonal-naive",
      lagMs: 7 * dayMs,
    });

    return yield* comparePerformance(model, [lastValue, sameDayLastWeek], {
      metrics: ["mae"],
      aggregation: { kind: "overall" },
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

if (comparison.model.kind === "overall") {
  console.log("effect-prophet", comparison.model.bucket.scores[0]?.value.toFixed(2));
}

for (const { baseline, metrics } of comparison.baselines) {
  if (metrics.kind === "overall") {
    console.log(baseline.kind, metrics.bucket.scores[0]?.value.toFixed(2));
  }
}

```

```txt [Output]
effect-prophet 2.06
last-observation 6.34
seasonal-naive 4.23
```

If a simple rule does about as well as the model, the model isn't adding much.

## A final, honest check

If you try many settings, the winner can look better than it really is, simply because it got
lucky on those particular rounds. To get an honest score, lock away the most recent data before
you start, choose settings using only the rest, then test the winner once on the locked-away
part.

```ts
import { Effect } from "effect";
import { evaluateHoldout, prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const all = Array.from({ length: 140 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 3 * Math.sin(day * 12.9898),
}));

// Lock away the last two weeks. The search never sees them.
const development = all.slice(0, 126);

const holdout = all.slice(126);

const report = await Effect.runPromise(
  Effect.gen(function* () {
    const search = yield* searchModels(development, {
      candidates: [
        { id: "default", options: {} },
        { id: "flexible-trend", options: { map: { changepointPriorScale: 0.5 } } },
      ],
      plan: {
        horizonMs: 14 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 14 * dayMs },
      },
      objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
      failurePolicy: "record",
    });

    const winner = search.candidates[search.selected.candidateIndex];

    if (winner?.kind !== "success") {
      return yield* Effect.die(new Error("The search did not select a successful candidate"));
    }

    // Fit the winner once on all development data and score it on the locked-away weeks.
    return yield* evaluateHoldout({
      development,
      holdout,
      search,
      selectedCandidate: {
        id: winner.id,
        candidateIndex: winner.candidateIndex,
        options: winner.options,
        score: winner.score,
      },
      metrics: { metrics: ["mae", "rmse"], aggregation: { kind: "overall" } },
      baselines: [{ kind: "last-observation" }],
      provenance: { build: "docs-example", environment: "node" },
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

console.log("selected:", report.selected.id);

console.log(JSON.stringify(report.metrics, null, 2));

```

`evaluateHoldout` fits the selected settings once on all the development data and scores them
on the held-out rows. The report is plain data, so you can save it with
`encodeEvaluationReport` and load it again with `decodeEvaluationReport`. Loading recomputes the
scores, so a report saved by an earlier version of Effect Prophet may be rejected with an
`EvaluationReportError`. Rerun the check after upgrading.