---
url: https://jer-k.github.io/effect-prophet/examples/residuals.md
description: >-
  Check a fitted model with residuals, and check whether the L-BFGS optimizer
  stopped before the best fit.
---

# Checking the fit

How well does a model follow your data? `residuals` predicts each of your measurements with a
fitted model and returns how far off it was. A good fit leaves residuals that bounce around zero;
a pattern in them shows something the model missed.

```ts
import { Effect } from "effect";
import { fit, prophetFittingBackendLayer, residuals } from "effect-prophet";

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

// Sales grow steadily, then jump on day 60 and stay higher.
const history = Array.from({ length: 90 }, (_, day) => ({
  timestamp: date(day),
  value:
    100 +
    day * 0.5 +
    10 * Math.sin((2 * Math.PI * day) / 7) +
    4 * Math.sin(day * 12.9898) +
    (day >= 60 ? 15 : 0),
}));

const rows = await Effect.runPromise(
  Effect.gen(function* () {
    const model = yield* fit(history, {
      map: { changepoints: { mode: "explicit", timestamps: [] } },
    });

    return yield* residuals(model, history);
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

// One point per day, ready for any charting library.
const points = rows.map((row) => ({
  date: new Date(row.timestamp).toISOString().slice(0, 10),
  residual: Number(row.residual.toFixed(1)),
}));

console.log(points.slice(0, 3));

// A straight trend can't follow the jump, so the residuals drift instead of centering on zero.
for (const start of [0, 30, 60]) {
  const block = rows.slice(start, start + 30);
  const mean = block.reduce((total, row) => total + row.residual, 0) / block.length;

  console.log(`days ${start}-${start + 29}: mean residual ${mean.toFixed(1)}`);
}

```

```txt [Output]
[
  { date: '2025-01-01', residual: 4.2 },
  { date: '2025-01-02', residual: 5.8 },
  { date: '2025-01-03', residual: 7.1 }
]
days 0-29: mean residual 1.2
days 30-59: mean residual -5.0
days 60-89: mean residual 3.7
```

This model has no changepoints, so its straight trend can't follow the jump on day 60. The
residuals show it: the model predicts too high in the weeks before the jump, so they turn negative,
and too low after it. Fitted with Prophet's automatic changepoints instead, the trend bends toward
the jump and the drift shrinks to under 2 either way.

## What you get back

One row per measurement, sorted by date. Measurements with the same timestamp keep the order you
passed them in.

| Field       | Meaning                                                               |
| ----------- | --------------------------------------------------------------------- |
| `actual`    | The value you measured                                                |
| `predicted` | The model's forecast for that moment, exactly as `predict` returns it |
| `residual`  | `actual - predicted`: positive when the model predicted too low       |

Each measurement is predicted from its own row, so it needs whatever the model needs to forecast
it: a `capacity` for growth with a ceiling, regressor values, and conditions for conditional
seasonalities. Rows are never matched by timestamp, so two measurements on the same date with
different regressor values each get their own prediction.

`residuals` never refits the model, and doesn't need a fitting backend. If a residual is too large
to represent, which only happens with values near the limits of a JavaScript number, it fails with
a `PredictionError` whose reason is `non-finite-forecast`.

## Measurements the model hasn't seen

Residuals on the history a model was fitted on are always a little flattering: the model has
already adjusted to those values. To see how it does on new data, hold some back:

```ts
import { Effect } from "effect";
import { fit, prophetFittingBackendLayer, residuals } from "effect-prophet";

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

const sales = Array.from({ length: 104 }, (_, day) => ({
  timestamp: date(day),
  value: 100 + day * 0.5 + 10 * Math.sin((2 * Math.PI * day) / 7) + 4 * Math.sin(day * 12.9898),
}));

// Hold back the last two weeks; the model never sees them.
const training = sales.slice(0, 90);

const heldOut = sales.slice(90);

const meanAbsolute = (rows: ReadonlyArray<{ readonly residual: number }>) =>
  rows.reduce((total, row) => total + Math.abs(row.residual), 0) / rows.length;

const [inSample, outOfSample] = await Effect.runPromise(
  Effect.gen(function* () {
    const model = yield* fit(training);

    return [yield* residuals(model, training), yield* residuals(model, heldOut)] as const;
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

console.log(`training days: mean absolute residual ${meanAbsolute(inSample).toFixed(2)}`);

console.log(`held-out days: mean absolute residual ${meanAbsolute(outOfSample).toFixed(2)}`);

```

```txt [Output]
training days: mean absolute residual 2.47
held-out days: mean absolute residual 2.64
```

`residuals` can't tell which measurements the model was trained on; that's up to you. For a
proper accuracy estimate over several periods, use
[cross-validation](./cross-validation).

## Did the optimizer finish?

For histories of 100 rows or more, `fit` uses the L-BFGS optimizer, as Python Prophet does. On
some histories, often ones with outliers or growth that changes sharply, it stops before it
reaches the best fit. Python Prophet 1.4.0 stops short on the same histories, by about as much,
but not at exactly the same point, so the two libraries agree less closely there.

To check a fit, take one Newton step from where it stopped:

```ts
import { Effect } from "effect";
import {
  fit,
  isMcmcModel,
  prophetFittingBackendLayer,
  warmStartParameters,
  type FittedProphet,
} from "effect-prophet";

const date = (day: number) => new Date(Date.UTC(2025, 0, 1 + day)).toISOString();

// A year of daily sales: growth that slows mid-year, a weekly pattern and four one-off spikes.
const sales = Array.from({ length: 365 }, (_, day) => ({
  timestamp: date(day),
  value:
    100 +
    (day < 180 ? day * 0.5 : 90 - (day - 180) * 0.2) +
    10 * Math.sin((2 * Math.PI * day) / 7) +
    3 * Math.sin(day * 12.9898) +
    ([50, 120, 230, 300].includes(day) ? 40 : 0),
}));

const report = await Effect.runPromise(
  Effect.gen(function* () {
    const model = yield* fit(sales);
    const warmStart = warmStartParameters(model);

    // One Newton step from where the fit stopped, on the same history.
    const step = yield* fit(sales, {
      map: { warmStart, optimizer: { algorithm: "newton", maxIterations: 1 } },
    });

    // Finish the fit with Newton, starting from the same place.
    const finished = yield* fit(sales, { map: { warmStart, optimizer: { algorithm: "newton" } } });

    return { model, step, finished };
  }).pipe(Effect.provide(prophetFittingBackendLayer)),
);

const { model, step, finished } = report;

// Models fitted with `mcmc` report a sampler instead of an optimizer.
const objective = (fitted: FittedProphet) =>
  isMcmcModel(fitted) ? Number.NaN : fitted.fitSummary.objective;

console.log(`stopped by:      ${model.fitSummary.termination}`);

console.log(`one step gains:  ${(objective(model) - objective(step)).toFixed(3)}`);

console.log(`Newton gains:    ${(objective(model) - objective(finished)).toFixed(3)}`);

```

```txt [Output]
stopped by:      relative-gradient
one step gains:  0.418
Newton gains:    0.713
```

* `fitSummary.objective` measures how badly the model fits its history and priors: lower is
  better. Compare it only between models fitted to the same history with the same settings.
* A gain above `0.01` means L-BFGS stopped early. In our tests, every fit that stopped more than
  1 short of the best gained more than `0.02` from one step. Fits that finished usually gained
  less than `0.0001`, though a few that stopped within `0.1` of the best gained up to `0.03`.
  Finishing the fit with Newton from the same `warmStart`, as above, came within `0.2` of the best
  fit every time.
* `fitSummary.termination` can't tell you this. Fits that stop early end with the same reasons,
  usually `relative-gradient`, as fits that finish. Tighter L-BFGS tolerances in
  `map.optimizer.lbfgs` rarely help: the optimizer is creeping along, not giving up too soon.
* Stopping early rarely matters much for fits from scratch. In our tests it left the objective up
  to 22 short, which moved fitted values by at most a tenth of the history's standard deviation,
  and usually by less than 4%. A warm start after many new rows can do far worse; see
  [Updating a model with new data](./saving-models#updating-a-model-with-new-data).
* The check costs one Newton step: 1 to 50 milliseconds on our histories of up to six years of
  daily data. Newton itself is much slower than L-BFGS: finishing a fit, or fitting from scratch
  with Newton, took 2 to more than 100 times as long as the L-BFGS fit. Check first, and finish
  only the fits that need it. Newton is also subject to [a memory limit](../python/differences)
  on very large models.
* `fit` keeps L-BFGS and its tolerances by default so that results match Python Prophet's.