Choosing settings

The defaults work well for a lot of data, but you can often do better by trying a few settings and keeping the one that forecasts best.

Benchmarked against Python Prophet

Settings search and the final holdout check are benchmarked on all three trend types. See the cross-validation and choosing settings benchmarks.

Comparing settings

searchModels runs cross-validation for each set of options you give it, on exactly the same rounds, and picks the one with the lowest error.

import { Effect } from "effect";
import { prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const history = Array.from({ length: 120 }, (_, day) => ({
  timestamp: date(day),
  value: (day < 70 ? 50 + day * 0.2 : 64 + (day - 70) * 1.2) + 3 * Math.sin(day * 12.9898),
}));

const result = await Effect.runPromise(
  searchModels(history, {
    // Each candidate is a set of fit options you want to try.
    candidates: [
      { id: "stiff-trend", options: { map: { changepointPriorScale: 0.001 } } },
      { id: "default-trend", options: { map: { changepointPriorScale: 0.05 } } },
      { id: "flexible-trend", options: { map: { changepointPriorScale: 0.5 } } },
    ],
    plan: {
      horizonMs: 7 * dayMs,
      cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
    },
    objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
    // "record" keeps going if one candidate fails; "fail-fast" stops at the first failure.
    failurePolicy: "record",
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

for (const candidate of result.candidates) {
  if (candidate.kind === "success") {
    console.log(candidate.id, candidate.score.toFixed(3));
  } else {
    console.log(candidate.id, "failed:", candidate.failure.tag);
  }
}

console.log("best:", result.selected.id);

txt

stiff-trend 12.556
default-trend 3.459
flexible-trend 2.907
best: flexible-trend

This data has a sudden change in growth, so the most flexible trend wins. Good settings to try first:

changepointPriorScale: try 0.01, 0.05, 0.1, 0.5
seasonalityPriorScale: try 0.01, 0.1, 1, 10
holidaysPriorScale: try 0.01, 0.1, 1, 10, if you have events, country holidays or regressors
seasonalityMode: "additive" or "multiplicative"

seasonalityPriorScale and holidaysPriorScale set the prior for every seasonality, or every event, country holiday and regressor, that doesn't set its own priorScale. Lower values keep those effects small. As in Python Prophet, regressors follow holidaysPriorScale, so give each regressor its own priorScale if you want to tune holidays alone.

You can compare up to 32 candidates in one search (maximumSearchCandidates).

To pick the winner by a score of your own, pass a custom metric as objective.metric. The final holdout check below needs a built-in metric.

Trying every combination

Rather than writing out each candidate, give makeParameterGrid the options every candidate shares and the values to try for each setting. It returns one candidate per combination, ready for searchModels:

import { Effect } from "effect";
import { makeParameterGrid, prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

// Growth speeds up on day 70, and a weekly pattern grows with the trend.
const history = Array.from({ length: 120 }, (_, day) => {
  const trend = day < 70 ? 50 + day * 0.2 : 64 + (day - 70) * 1.2;

  return {
    timestamp: date(day),
    value: trend * (1 + 0.1 * Math.sin((2 * Math.PI * day) / 7)) + Math.sin(day * 12.9898),
  };
});

const result = await Effect.runPromise(
  Effect.gen(function* () {
    // Every combination of the values below: 3 × 2 = 6 candidates.
    const candidates = yield* makeParameterGrid({
      base: { builtInSeasonalities: { daily: "off", yearly: "off" } },
      axes: {
        changepointPriorScale: [0.01, 0.1, 0.5],
        seasonalityMode: ["additive", "multiplicative"],
      },
    });

    return yield* searchModels(history, {
      candidates,
      plan: {
        horizonMs: 7 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
      },
      objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
      failurePolicy: "record",
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

for (const candidate of result.candidates) {
  if (candidate.kind === "success") {
    const { map, seasonalityMode } = candidate.options;

    console.log(
      candidate.id,
      map?.changepointPriorScale,
      seasonalityMode,
      candidate.score.toFixed(3),
    );
  }
}

console.log("best:", result.selected.id);

txt

grid-0-0 0.01 additive 7.236
grid-0-1 0.01 multiplicative 5.006
grid-1-0 0.1 additive 2.946
grid-1-1 0.1 multiplicative 1.822
grid-2-0 0.5 additive 2.773
grid-2-1 0.5 multiplicative 1.685
best: grid-2-1

These settings can be varied:

Setting	Replaces	Example values
`changepointPriorScale`	`map.changepointPriorScale`	`[0.01, 0.1, 0.5]`
`changepoints`	`map.changepoints`	`[{ mode: "auto", range: 0.8 }, { mode: "auto", range: 0.95 }]`
`seasonalityPriorScale`	`seasonalityPriorScale`	`[0.01, 0.1, 1, 10]`
`holidaysPriorScale`	`holidaysPriorScale`	`[0.01, 0.1, 1, 10]`
`seasonalityMode`	`seasonalityMode`	`["additive", "multiplicative"]`
`holidaysMode`	`holidaysMode`	`["additive", "multiplicative"]`
`builtInSeasonalities`	`builtInSeasonalities`	`[{ yearly: { mode: "on", priorScale: 1 } }, {}]`
`seasonalities`	`seasonalities`	`[[], [{ name: "monthly", periodDays: 30.5, fourierOrder: 5 }]]`
`scaling`	`scaling`	`["absmax", "minmax"]`

Each value replaces the base's setting outright. A list or object, such as a set of seasonalities, counts as one value; it isn't combined any further. To try one seasonality's own prior scale instead of seasonalityPriorScale, give each version of the whole seasonalities list as a value.

Candidates come out in the order of the table, with the first setting changing slowest, whatever order you write the axes in. Each ID is grid followed by the position of its value in each setting you gave: grid-2-1 above is the third changepointPriorScale and the second seasonalityMode. With no axes, the one candidate is the base, with the ID grid.

makeParameterGrid checks the whole grid before you start a search, and fails with a SearchOptionsError when:

the grid has more than 32 combinations (reason is limit-exceeded, checked before any value is read),
a setting has the same value twice (duplicate-choice), or has no values or isn't in the table (invalid-shape),
the base options are invalid (invalid-base-options; the path starts with base),
a combination is invalid (invalid-candidate-options, with the combination's position in candidateIndex). When the problem is in one value, the path points at it, such as ["axes", "changepointPriorScale", 1]. When it comes from how values combine, such as a seasonality named like a base regressor, the path points at the candidate: ["candidates", 3, "options", ...].

For invalid base or combination options, cause holds the original OptionsError.

It fits nothing and needs no fitting backend.

Comparing against simple rules

Before trusting a model, check that it beats the obvious guesses:

Last observation: tomorrow will be the same as today.
Training mean: tomorrow will be the average of everything so far.
Seasonal naive: tomorrow will be the same as this day last week.

import { Effect } from "effect";
import {
  comparePerformance,
  crossValidate,
  crossValidateBaseline,
  prophetFittingBackendLayer,
} from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const history = Array.from({ length: 120 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 3 * Math.sin(day * 12.9898),
}));

const comparison = await Effect.runPromise(
  Effect.gen(function* () {
    const model = yield* crossValidate(
      history,
      {},
      {
        horizonMs: 7 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
      },
    );

    // Simple rules of thumb, checked on exactly the same days as the model.
    const lastValue = yield* crossValidateBaseline(history, model.plan, {
      kind: "last-observation",
    });

    const sameDayLastWeek = yield* crossValidateBaseline(history, model.plan, {
      kind: "seasonal-naive",
      lagMs: 7 * dayMs,
    });

    return yield* comparePerformance(model, [lastValue, sameDayLastWeek], {
      metrics: ["mae"],
      aggregation: { kind: "overall" },
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

if (comparison.model.kind === "overall") {
  console.log("effect-prophet", comparison.model.bucket.scores[0]?.value.toFixed(2));
}

for (const { baseline, metrics } of comparison.baselines) {
  if (metrics.kind === "overall") {
    console.log(baseline.kind, metrics.bucket.scores[0]?.value.toFixed(2));
  }
}

txt

effect-prophet 2.06
last-observation 6.34
seasonal-naive 4.23

If a simple rule does about as well as the model, the model isn't adding much.

A final, honest check

If you try many settings, the winner can look better than it really is, simply because it got lucky on those particular rounds. To get an honest score, lock away the most recent data before you start, choose settings using only the rest, then test the winner once on the locked-away part.

import { Effect } from "effect";
import { evaluateHoldout, prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const all = Array.from({ length: 140 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 3 * Math.sin(day * 12.9898),
}));

// Lock away the last two weeks. The search never sees them.
const development = all.slice(0, 126);

const holdout = all.slice(126);

const report = await Effect.runPromise(
  Effect.gen(function* () {
    const search = yield* searchModels(development, {
      candidates: [
        { id: "default", options: {} },
        { id: "flexible-trend", options: { map: { changepointPriorScale: 0.5 } } },
      ],
      plan: {
        horizonMs: 14 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 14 * dayMs },
      },
      objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
      failurePolicy: "record",
    });

    const winner = search.candidates[search.selected.candidateIndex];

    if (winner?.kind !== "success") {
      return yield* Effect.die(new Error("The search did not select a successful candidate"));
    }

    // Fit the winner once on all development data and score it on the locked-away weeks.
    return yield* evaluateHoldout({
      development,
      holdout,
      search,
      selectedCandidate: {
        id: winner.id,
        candidateIndex: winner.candidateIndex,
        options: winner.options,
        score: winner.score,
      },
      metrics: { metrics: ["mae", "rmse"], aggregation: { kind: "overall" } },
      baselines: [{ kind: "last-observation" }],
      provenance: { build: "docs-example", environment: "node" },
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

console.log("selected:", report.selected.id);

console.log(JSON.stringify(report.metrics, null, 2));

evaluateHoldout fits the selected settings once on all the development data and scores them on the held-out rows. The report is plain data, so you can save it with encodeEvaluationReport and load it again with decodeEvaluationReport. Loading recomputes the scores, so a report saved by an earlier version of Effect Prophet may be rejected with an EvaluationReportError. Rerun the check after upgrading.

By trend

By feature

Choosing settings

Comparing settings

Trying every combination

Comparing against simple rules

A final, honest check

Choosing settings ​

Comparing settings ​

Trying every combination ​

Comparing against simple rules ​

A final, honest check ​

Choosing settings

Comparing settings

Trying every combination

Comparing against simple rules

A final, honest check