Choosing settings

The defaults work well for a lot of data, but you can often do better by trying a few settings and keeping the one that forecasts best.

Benchmarked against Python Prophet

Settings search and the final holdout check are benchmarked on all three trend types. See the cross-validation and choosing settings benchmarks.

Comparing settings

searchModels runs cross-validation for each set of options you give it, on exactly the same rounds, and picks the one with the lowest error.

import { Effect } from "effect";
import { prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const history = Array.from({ length: 120 }, (_, day) => ({
  timestamp: date(day),
  value: (day < 70 ? 50 + day * 0.2 : 64 + (day - 70) * 1.2) + 3 * Math.sin(day * 12.9898),
}));

const result = await Effect.runPromise(
  searchModels(history, {
    // Each candidate is a set of fit options you want to try.
    candidates: [
      { id: "stiff-trend", options: { map: { changepointPriorScale: 0.001 } } },
      { id: "default-trend", options: { map: { changepointPriorScale: 0.05 } } },
      { id: "flexible-trend", options: { map: { changepointPriorScale: 0.5 } } },
    ],
    plan: {
      horizonMs: 7 * dayMs,
      cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
    },
    objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
    // "record" keeps going if one candidate fails; "fail-fast" stops at the first failure.
    failurePolicy: "record",
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

for (const candidate of result.candidates) {
  if (candidate.kind === "success") {
    console.log(candidate.id, candidate.score.toFixed(3));
  } else {
    console.log(candidate.id, "failed:", candidate.failure.tag);
  }
}

console.log("best:", result.selected.id);

stiff-trend 12.556
default-trend 3.459
flexible-trend 2.907
best: flexible-trend

This data has a sudden change in growth, so the most flexible trend wins. Good settings to try first:

• changepointPriorScale: try 0.01, 0.05, 0.1, 0.5
• seasonalityMode: "additive" or "multiplicative"
• priorScale on each seasonality: try 0.1, 1, 10

You can compare up to 32 candidates in one search.

Comparing against simple rules

Before trusting a model, check that it beats the obvious guesses:

• Last observation: tomorrow will be the same as today.
• Training mean: tomorrow will be the average of everything so far.
• Seasonal naive: tomorrow will be the same as this day last week.

import { Effect } from "effect";
import {
  comparePerformance,
  crossValidate,
  crossValidateBaseline,
  prophetFittingBackendLayer,
} from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const history = Array.from({ length: 120 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 3 * Math.sin(day * 12.9898),
}));

const comparison = await Effect.runPromise(
  Effect.gen(function* () {
    const model = yield* crossValidate(
      history,
      {},
      {
        horizonMs: 7 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 7 * dayMs },
      },
    );

    // Simple rules of thumb, checked on exactly the same days as the model.
    const lastValue = yield* crossValidateBaseline(history, model.plan, {
      kind: "last-observation",
    });

    const sameDayLastWeek = yield* crossValidateBaseline(history, model.plan, {
      kind: "seasonal-naive",
      lagMs: 7 * dayMs,
    });

    return yield* comparePerformance(model, [lastValue, sameDayLastWeek], {
      metrics: ["mae"],
      aggregation: { kind: "overall" },
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

if (comparison.model.kind === "overall") {
  console.log("effect-prophet", comparison.model.bucket.scores[0]?.value.toFixed(2));
}

for (const { baseline, metrics } of comparison.baselines) {
  if (metrics.kind === "overall") {
    console.log(baseline.kind, metrics.bucket.scores[0]?.value.toFixed(2));
  }
}

effect-prophet 2.06
last-observation 6.34
seasonal-naive 4.23

If a simple rule does about as well as the model, the model isn't adding much.

A final, honest check

If you try many settings, the winner can look better than it really is, simply because it got lucky on those particular rounds. To get an honest score, lock away the most recent data before you start, choose settings using only the rest, then test the winner once on the locked-away part.

import { Effect } from "effect";
import { evaluateHoldout, prophetFittingBackendLayer, searchModels } from "effect-prophet";

const dayMs = 86_400_000;

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const all = Array.from({ length: 140 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 3 * Math.sin(day * 12.9898),
}));

// Lock away the last two weeks. The search never sees them.
const development = all.slice(0, 126);

const holdout = all.slice(126);

const report = await Effect.runPromise(
  Effect.gen(function* () {
    const search = yield* searchModels(development, {
      candidates: [
        { id: "default", options: {} },
        { id: "flexible-trend", options: { map: { changepointPriorScale: 0.5 } } },
      ],
      plan: {
        horizonMs: 14 * dayMs,
        cutoffs: { mode: "generated", initialMs: 60 * dayMs, periodMs: 14 * dayMs },
      },
      objective: { metric: "mae", aggregation: { kind: "overall" }, direction: "minimize" },
      failurePolicy: "record",
    });

    const winner = search.candidates[search.selected.candidateIndex];

    if (winner?.kind !== "success") {
      return yield* Effect.die(new Error("The search did not select a successful candidate"));
    }

    // Fit the winner once on all development data and score it on the locked-away weeks.
    return yield* evaluateHoldout({
      development,
      holdout,
      search,
      selectedCandidate: {
        id: winner.id,
        candidateIndex: winner.candidateIndex,
        options: winner.options,
        score: winner.score,
      },
      metrics: { metrics: ["mae", "rmse"], aggregation: { kind: "overall" } },
      baselines: [{ kind: "last-observation" }],
      provenance: { build: "docs-example", environment: "node" },
    });
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

console.log("selected:", report.selected.id);

console.log(JSON.stringify(report.metrics, null, 2));

evaluateHoldout fits the selected settings once on all the development data and scores them on the held-out rows. The report is plain data, so you can save it with encodeEvaluationReport and load it again with decodeEvaluationReport.