Cross-validation, eight months

match ← All flat growth benchmarks

Two rolling cutoffs on the 256-day history, with rolling-window metrics.

Input

benchmark/inputs/v1/flat-mixed-components-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 2.4e-7 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-08-13	35.0253	35.0005	35.0005	1.6e-7
2020-08-14	48.7637	48.6525	48.6525	1.8e-7
2020-08-15	41.0876	41.0662	41.0662	2.4e-7
2020-08-16	28.8756	28.9251	28.9251	2.4e-7
2020-08-17	21.4046	21.3585	21.3585	1.8e-7
2020-08-18	36.2570	36.1993	36.1993	1.6e-7
2020-08-19	34.9214	34.9960	34.9960	7.3e-8
2020-08-20	45.8771	45.9534	45.9534	4.3e-8
2020-08-21	48.6894	48.6567	48.6567	9.5e-9
2020-08-22	41.0568	41.0671	41.0671	9.2e-8
2020-08-23	34.8913	35.0009	35.0009	4.9e-8
2020-08-24	22.5415	22.5599	22.5599	1.2e-7
2020-08-25	24.1269	24.0527	24.0527	4.7e-8
2020-08-26	34.9766	34.9930	34.9930	2.7e-8

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	118 µs	511 µs	Not compared: different work
Plan cross-validation folds	2.27 ms	768 µs	Not compared: different work
Cross-validate	6.27 ms	39.0 ms	Not compared: different work
Compute metrics	85 µs	5.48 ms	Not compared: different work
Score baselines	2.71 ms	1.40 ms	Not compared: different work
First cross-validation in a new process	192 ms	1.47 s	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-08-12T00:00:00.000Z",
      "2020-08-19T00:00:00.000Z"
    ]
  },
  "metricAggregation": "rolling"
}