Cross-validation and metrics

match ← All flat growth benchmarks

Three rolling cutoffs, a 7-day horizon, all metrics and a baseline comparison.

Input

benchmark/inputs/v1/flat-mixed-components.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 2.8e-7 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-04	35.0775	34.9780	34.9780	1.2e-7
2020-03-05	45.8872	45.9405	45.9405	3.0e-8
2020-03-06	34.9423	35.0019	35.0019	1.0e-7
2020-03-07	41.1218	41.0645	41.0645	7.7e-9
2020-03-08	28.9133	28.8939	28.8939	1.7e-7
2020-03-09	22.4365	22.5775	22.5775	2.8e-7
2020-03-10	24.0283	24.0826	24.0826	2.6e-7
2020-03-11	35.0568	35.0021	35.0021	1.0e-7
2020-03-12	45.9078	45.9398	45.9398	9.8e-8
2020-03-13	48.5866	48.6775	48.6775	2.6e-8
2020-03-14	41.1455	41.0759	41.0759	1.0e-8
2020-03-15	30.2075	30.0881	30.0881	4.7e-8
2020-03-16	34.9665	35.0021	35.0021	1.0e-7
2020-03-17	24.0576	24.0795	24.0795	1.9e-7
2020-03-18	35.1048	34.9897	34.9897	3.0e-9
2020-03-19	45.9668	45.9330	45.9330	8.4e-8
2020-03-20	48.5708	48.6675	48.6675	5.7e-8
2020-03-21	36.2070	36.2037	36.2037	5.4e-8
2020-03-22	28.9741	28.9071	28.9071	1.7e-7
2020-03-23	21.2751	21.3559	21.3559	2.0e-7
2020-03-24	23.9598	24.0746	24.0746	1.2e-7

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	68 µs	386 µs	5.7× faster
Plan cross-validation folds	1.49 ms	741 µs	2.0× slower
Cross-validate	6.50 ms	46.1 ms	7.1× faster
Compute metrics	218 µs	5.34 ms	24× faster
Score baselines	1.28 ms	1.46 ms	1.1× faster
First cross-validation in a new process	185 ms	1.47 s	7.9× faster

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-03T00:00:00.000Z",
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "mapeZeroActual": "exclude"
}