Cross-validation with a forecast regressor and ranges

match ← All flat growth benchmarks

The same folds with simulated futures that include the temperature model's draws.

Input

benchmark/inputs/v4/flat-correlated-regressors.json: 730 history rows and 180 forecast rows.

Output

Largest difference: 0.012 (0.094% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2021-12-03	45.5639	46.0110	46.0220	0.011
2021-12-04	43.8466	44.6465	44.6576	0.011
2021-12-05	43.2507	43.1670	43.1783	0.011
2021-12-06	43.8873	42.6254	42.6368	0.011
2021-12-07	44.4927	43.3799	43.3915	0.012
2021-12-08	45.2585	44.8417	44.8539	0.012
2021-12-09	46.2383	45.8664	45.8788	0.012
2021-12-17	45.5393	45.2172	45.2096	0.0076
2021-12-18	43.9466	43.8397	43.8323	0.0074
2021-12-19	42.8487	42.3669	42.3597	0.0071
2021-12-20	43.5652	41.8428	41.8358	0.0070
2021-12-21	44.6979	42.5891	42.5822	0.0069
2021-12-22	45.2593	44.0291	44.0213	0.0078
2021-12-23	45.8218	45.0435	45.0356	0.0079

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Cross-validate with ranges	61.8 ms	291 ms	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Cross-validate with ranges: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "yearly-custom",
        "periodDays": 365.25,
        "fourierOrder": 10,
        "priorScale": 10
      },
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": [
      {
        "name": "temperature",
        "priorScale": 10,
        "standardization": "auto",
        "predictor": true
      },
      {
        "name": "baseline",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2021-12-02T00:00:00.000Z",
      "2021-12-16T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "interval": {
    "seed": 19,
    "samples": 128,
    "intervalWidth": 0.8
  }
}