Cross-validation and metrics

match ← All linear growth benchmarks

Two rolling cutoffs, a 7-day horizon, all metrics and a baseline comparison.

Input

benchmark/inputs/v1/linear-offset-scaling.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 2.9e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-11	107.2568	107.1688	107.1688	2.9e-11
2020-03-12	107.4776	107.4962	107.4962	2.9e-11
2020-03-13	107.6213	107.7018	107.7018	2.8e-11
2020-03-14	107.7396	107.6554	107.6554	2.8e-11
2020-03-15	107.6534	107.5295	107.5295	2.7e-11
2020-03-16	107.5228	107.5532	107.5532	2.8e-11
2020-03-17	107.7277	107.7317	107.7317	2.8e-11
2020-03-18	108.1448	108.0262	108.0262	1.8e-11
2020-03-19	108.3767	108.3442	108.3442	1.8e-11
2020-03-20	108.4456	108.5444	108.5444	1.8e-11
2020-03-21	108.5155	108.5134	108.5134	1.7e-11
2020-03-22	108.4600	108.3912	108.3912	1.7e-11
2020-03-23	108.3204	108.4014	108.4014	1.8e-11
2020-03-24	108.4700	108.5823	108.5823	1.8e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	12 µs	402 µs	35× faster
Plan cross-validation folds	1.17 ms	659 µs	1.8× slower
Cross-validate	9.52 ms	75.7 ms	8.0× faster
Compute metrics	214 µs	5.40 ms	25× faster
Score baselines	1.08 ms	1.10 ms	1.0× faster
First cross-validation in a new process	224 ms	1.48 s	6.6× faster

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "overall",
  "mapeZeroActual": "exclude"
}