Cross-validation, eight months

match ← All logistic growth benchmarks

Two rolling cutoffs on the 256-day history, with rolling-window metrics.

Input

benchmark/inputs/v1/logistic-implicit-floor-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 0.028 (0.034% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-08-13	87.3514	87.3331	87.3137	0.019
2020-08-14	87.6488	87.5376	87.5413	0.0037
2020-08-15	87.4249	87.4024	87.4110	0.0086
2020-08-16	87.0921	87.1415	87.1515	0.010
2020-08-17	87.0732	87.0283	87.0564	0.028
2020-08-18	87.2845	87.2244	87.2453	0.021
2020-08-19	87.6215	87.6939	87.6790	0.015
2020-08-20	88.1043	88.1796	88.1595	0.020
2020-08-21	88.4210	88.3871	88.3894	0.0024
2020-08-22	88.2403	88.2486	88.2558	0.0072
2020-08-23	87.8794	87.9853	87.9940	0.0086
2020-08-24	87.8558	87.8750	87.9012	0.026
2020-08-25	88.1455	88.0707	88.0902	0.019
2020-08-26	88.5220	88.5367	88.5214	0.015

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	31 µs	500 µs	Not compared: different work
Plan cross-validation folds	1.32 ms	718 µs	Not compared: different work
Cross-validate	14.1 ms	41.5 ms	Not compared: different work
Compute metrics	62 µs	5.31 ms	Not compared: different work
Score baselines	2.15 ms	1.30 ms	Not compared: different work
First cross-validation in a new process	190 ms	1.45 s	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "logistic",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-08-12T00:00:00.000Z",
      "2020-08-19T00:00:00.000Z"
    ]
  },
  "metricAggregation": "rolling"
}