Cross-validation with a forecast regressor and ranges

match ← All linear growth benchmarks

The same folds with simulated futures that include the regressor model's draws.

Input

benchmark/inputs/v1/map-regressors-medium.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 0.0024 (0.0025% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-08-23	97.5341	97.6352	97.6368	0.0016
2020-08-24	98.5406	98.5658	98.5679	0.0020
2020-08-25	97.3639	97.3234	97.3254	0.0021
2020-08-26	97.8038	97.8828	97.8849	0.0020
2020-08-27	98.3144	98.4353	98.4373	0.0020
2020-08-28	99.3311	99.3387	99.3411	0.0024
2020-08-29	99.8524	99.8764	99.8788	0.0024
2020-09-03	102.6061	102.8260	102.8260	7.3e-5
2020-09-04	101.7963	101.9156	101.9153	0.00022
2020-09-05	102.7016	102.7640	102.7638	0.00022
2020-09-06	103.0832	103.2447	103.2445	0.00017
2020-09-07	103.5750	103.7194	103.7193	0.00012
2020-09-08	104.1993	104.1885	104.1884	7.6e-5
2020-09-09	106.8323	106.8125	106.8124	0.00010

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Cross-validate with ranges	51.9 ms	255 ms	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Cross-validate with ranges: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [],
    "events": [],
    "regressors": [
      {
        "name": "binary-auto",
        "priorScale": 10,
        "standardization": "auto"
      },
      {
        "name": "numeric-auto",
        "priorScale": 10,
        "standardization": "auto",
        "predictor": true
      },
      {
        "name": "numeric-always",
        "priorScale": 7,
        "standardization": "always"
      },
      {
        "name": "numeric-never",
        "priorScale": 5,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-08-22T00:00:00.000Z",
      "2020-09-02T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "interval": {
    "seed": 19,
    "samples": 128,
    "intervalWidth": 0.8
  }
}