Cross-validation with a forecast regressor

match ← All flat growth benchmarks

Two cutoffs and a 7-day horizon, fitting the temperature model again in each fold. Python's cross_validation refits it on standardized values and drifts by about 15% here, so Python is fitted at each cutoff instead.

Input

benchmark/inputs/v4/flat-correlated-regressors.json: 730 history rows and 180 forecast rows.

Output

Largest difference: 0.012 (0.094% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2021-12-03	45.5639	46.0110	46.0220	0.011
2021-12-04	43.8466	44.6465	44.6576	0.011
2021-12-05	43.2507	43.1670	43.1783	0.011
2021-12-06	43.8873	42.6254	42.6368	0.011
2021-12-07	44.4927	43.3799	43.3915	0.012
2021-12-08	45.2585	44.8417	44.8539	0.012
2021-12-09	46.2383	45.8664	45.8788	0.012
2021-12-17	45.5393	45.2172	45.2096	0.0076
2021-12-18	43.9466	43.8397	43.8323	0.0074
2021-12-19	42.8487	42.3669	42.3597	0.0071
2021-12-20	43.5652	41.8428	41.8358	0.0070
2021-12-21	44.6979	42.5891	42.5822	0.0069
2021-12-22	45.2593	44.0291	44.0213	0.0078
2021-12-23	45.8218	45.0435	45.0356	0.0079

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	250 µs	750 µs	Not compared: different work
Plan cross-validation folds	2.61 ms	1.01 ms	Not compared: different work
Cross-validate	60.3 ms	227 ms	Not compared: different work
Compute metrics	98 µs	5.25 ms	Not compared: different work
Score baselines	7.10 ms	2.00 ms	Not compared: different work
First cross-validation in a new process	260 ms	1.64 s	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "yearly-custom",
        "periodDays": 365.25,
        "fourierOrder": 10,
        "priorScale": 10
      },
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": [
      {
        "name": "temperature",
        "priorScale": 10,
        "standardization": "auto",
        "predictor": true
      },
      {
        "name": "baseline",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2021-12-02T00:00:00.000Z",
      "2021-12-16T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}