Cross-validation and metrics

match ← All logistic growth benchmarks

One cutoff, a 7-day horizon, all metrics and a baseline comparison.

Input

benchmark/inputs/v1/logistic-implicit-floor.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 1.6e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-18	56.3482	56.2290	56.2290	1.6e-11
2020-03-19	57.1831	57.1500	57.1500	1.6e-11
2020-03-20	57.7257	57.8237	57.8237	1.5e-11
2020-03-21	58.1108	58.1079	58.1079	1.5e-11
2020-03-22	58.2935	58.2238	58.2238	1.4e-11
2020-03-23	58.4460	58.5260	58.5260	1.4e-11
2020-03-24	59.0232	59.1345	59.1345	1.4e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Convert input rows	17 µs	364 µs	22× faster
Plan cross-validation folds	971 µs	543 µs	1.8× slower
Cross-validate	16.6 ms	60.2 ms	3.6× faster
Compute metrics	192 µs	5.34 ms	28× faster
Score baselines	1.43 ms	787 µs	1.8× slower
First cross-validation in a new process	184 ms	1.53 s	8.3× faster

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "logistic",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "mapeZeroActual": "exclude"
}