Cross-validation and metrics

match ← All logistic growth benchmarks

One cutoff, a 7-day horizon, all metrics and a baseline comparison.

Input

benchmark/inputs/v1/logistic-implicit-floor.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 1.6e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-18	56.3482	56.2290	56.2290	1.6e-11
2020-03-19	57.1831	57.1500	57.1500	1.6e-11
2020-03-20	57.7257	57.8237	57.8237	1.5e-11
2020-03-21	58.1108	58.1079	58.1079	1.5e-11
2020-03-22	58.2935	58.2238	58.2238	1.4e-11
2020-03-23	58.4460	58.5260	58.5260	1.4e-11
2020-03-24	59.0232	59.1345	59.1345	1.4e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	10 µs	433 µs	Not compared: different work
Plan cross-validation folds	853 µs	573 µs	Not compared: different work
Cross-validate	17.1 ms	59.5 ms	Not compared: different work
Compute metrics	67 µs	5.16 ms	Not compared: different work
Score baselines	1.02 ms	769 µs	Not compared: different work
First cross-validation in a new process	191 ms	1.46 s	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "logistic",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}