Cross-validation with a forecast regressor

match ← All linear growth benchmarks

Two cutoffs and a 7-day horizon, fitting the regressor model again in each fold. Python's cross_validation refits it on standardized values, so Python is fitted at each cutoff instead.

Input

benchmark/inputs/v1/map-regressors-medium.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 0.0024 (0.0025% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-08-23	97.5341	97.6352	97.6368	0.0016
2020-08-24	98.5406	98.5658	98.5679	0.0020
2020-08-25	97.3639	97.3234	97.3254	0.0021
2020-08-26	97.8038	97.8828	97.8849	0.0020
2020-08-27	98.3144	98.4353	98.4373	0.0020
2020-08-28	99.3311	99.3387	99.3411	0.0024
2020-08-29	99.8524	99.8764	99.8788	0.0024
2020-09-03	102.6061	102.8260	102.8260	7.3e-5
2020-09-04	101.7963	101.9156	101.9153	0.00022
2020-09-05	102.7016	102.7640	102.7638	0.00022
2020-09-06	103.0832	103.2447	103.2445	0.00017
2020-09-07	103.5750	103.7194	103.7193	0.00012
2020-09-08	104.1993	104.1885	104.1884	7.6e-5
2020-09-09	106.8323	106.8125	106.8124	0.00010

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	108 µs	544 µs	Not compared: different work
Plan cross-validation folds	2.25 ms	796 µs	Not compared: different work
Cross-validate	51.6 ms	185 ms	Not compared: different work
Compute metrics	58 µs	5.50 ms	Not compared: different work
Score baselines	2.90 ms	1.42 ms	Not compared: different work
First cross-validation in a new process	245 ms	1.62 s	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [],
    "events": [],
    "regressors": [
      {
        "name": "binary-auto",
        "priorScale": 10,
        "standardization": "auto"
      },
      {
        "name": "numeric-auto",
        "priorScale": 10,
        "standardization": "auto",
        "predictor": true
      },
      {
        "name": "numeric-always",
        "priorScale": 7,
        "standardization": "always"
      },
      {
        "name": "numeric-never",
        "priorScale": 5,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-08-22T00:00:00.000Z",
      "2020-09-02T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}