Cross-validation with repeated timestamps

match ← All linear growth benchmarks

Folds over rows with repeated dates. Each repeated row is scored separately.

Input

benchmark/inputs/v1/training-duplicates.json: 96 history rows and 8 forecast rows.

Output

Largest difference: 1.2e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-01-19	14.8265	14.7488	14.7488	1.2e-11
2020-01-19	14.9419	14.7488	14.7488	1.2e-11
2020-01-19	14.9311	14.7488	14.7488	1.2e-11
2020-01-19	14.7976	14.7488	14.7488	1.2e-11
2020-01-20	15.0119	14.9200	14.9200	1.2e-11
2020-01-20	15.1112	14.9200	14.9200	1.2e-11
2020-01-20	15.1948	14.9200	14.9200	1.2e-11
2020-01-20	15.0932	14.9200	14.9200	1.2e-11
2020-01-21	15.2255	15.0911	15.0911	1.2e-11
2020-01-21	15.2441	15.0911	15.0911	1.2e-11
2020-01-21	15.3685	15.0911	15.0911	1.2e-11
2020-01-21	15.3270	15.0911	15.0911	1.2e-11
2020-01-20	15.0119	15.0738	15.0738	6.3e-12
2020-01-20	15.1112	15.0738	15.0738	6.3e-12
2020-01-20	15.1948	15.0738	15.0738	6.3e-12
2020-01-20	15.0932	15.0738	15.0738	6.3e-12
2020-01-21	15.2255	15.2980	15.2980	6.6e-12
2020-01-21	15.2441	15.2980	15.2980	6.6e-12
2020-01-21	15.3685	15.2980	15.2980	6.6e-12
2020-01-21	15.3270	15.2980	15.2980	6.6e-12
2020-01-22	15.5394	15.5221	15.5221	7.0e-12
2020-01-22	15.4659	15.5221	15.5221	7.0e-12
2020-01-22	15.5676	15.5221	15.5221	7.0e-12
2020-01-22	15.5810	15.5221	15.5221	7.0e-12

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Plan cross-validation folds	1.11 ms	945 µs	Not compared: different work
Cross-validate	6.85 ms	53.4 ms	Not compared: different work
Score baselines	968 µs	1.40 ms	Not compared: different work
Search settings	5.79 ms	56.6 ms	Not compared: different work
Final holdout check	3.85 ms	22.2 ms	Not compared: different work
Save report	236 µs	8 µs	Not compared: different work
Load report	219 µs	4 µs	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Plan cross-validation folds, Cross-validate, Score baselines, Search settings, Final holdout check, Save report, Load report: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-01-17T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [],
    "events": [],
    "regressors": []
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 259200000,
    "cutoffs": [
      "2020-01-18T00:00:00.000Z",
      "2020-01-19T00:00:00.000Z"
    ]
  },
  "metricAggregation": "overall",
  "candidates": [
    {
      "id": "control",
      "changepointPriorScale": 0.05
    }
  ],
  "holdoutRows": 8
}