Cross-validation and metrics

match ← All flat growth benchmarks

Three rolling cutoffs, a 7-day horizon, all metrics and a baseline comparison.

Input

benchmark/inputs/v1/flat-mixed-components.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 2.8e-7 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-04	35.0775	34.9780	34.9780	1.2e-7
2020-03-05	45.8872	45.9405	45.9405	3.0e-8
2020-03-06	34.9423	35.0019	35.0019	1.0e-7
2020-03-07	41.1218	41.0645	41.0645	7.7e-9
2020-03-08	28.9133	28.8939	28.8939	1.7e-7
2020-03-09	22.4365	22.5775	22.5775	2.8e-7
2020-03-10	24.0283	24.0826	24.0826	2.6e-7
2020-03-11	35.0568	35.0021	35.0021	1.0e-7
2020-03-12	45.9078	45.9398	45.9398	9.8e-8
2020-03-13	48.5866	48.6775	48.6775	2.6e-8
2020-03-14	41.1455	41.0759	41.0759	1.0e-8
2020-03-15	30.2075	30.0881	30.0881	4.7e-8
2020-03-16	34.9665	35.0021	35.0021	1.0e-7
2020-03-17	24.0576	24.0795	24.0795	1.9e-7
2020-03-18	35.1048	34.9897	34.9897	3.0e-9
2020-03-19	45.9668	45.9330	45.9330	8.4e-8
2020-03-20	48.5708	48.6675	48.6675	5.7e-8
2020-03-21	36.2070	36.2037	36.2037	5.4e-8
2020-03-22	28.9741	28.9071	28.9071	1.7e-7
2020-03-23	21.2751	21.3559	21.3559	2.0e-7
2020-03-24	23.9598	24.0746	24.0746	1.2e-7

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	52 µs	423 µs	Not compared: different work
Plan cross-validation folds	1.57 ms	769 µs	Not compared: different work
Cross-validate	4.50 ms	46.2 ms	Not compared: different work
Compute metrics	108 µs	5.44 ms	Not compared: different work
Score baselines	1.23 ms	1.47 ms	Not compared: different work
First cross-validation in a new process	190 ms	1.48 s	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-03T00:00:00.000Z",
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}