Cross-validation with repeated timestamps

match ← All linear growth benchmarks

Folds over rows with repeated dates. Each repeated row is scored separately.

Input

benchmark/inputs/v1/training-duplicates.json: 96 history rows and 8 forecast rows.

Output

Largest difference: 1.2e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-01-19	14.8265	14.7488	14.7488	1.2e-11
2020-01-19	14.9419	14.7488	14.7488	1.2e-11
2020-01-19	14.9311	14.7488	14.7488	1.2e-11
2020-01-19	14.7976	14.7488	14.7488	1.2e-11
2020-01-20	15.0119	14.9200	14.9200	1.2e-11
2020-01-20	15.1112	14.9200	14.9200	1.2e-11
2020-01-20	15.1948	14.9200	14.9200	1.2e-11
2020-01-20	15.0932	14.9200	14.9200	1.2e-11
2020-01-21	15.2255	15.0911	15.0911	1.2e-11
2020-01-21	15.2441	15.0911	15.0911	1.2e-11
2020-01-21	15.3685	15.0911	15.0911	1.2e-11
2020-01-21	15.3270	15.0911	15.0911	1.2e-11
2020-01-20	15.0119	15.0738	15.0738	6.3e-12
2020-01-20	15.1112	15.0738	15.0738	6.3e-12
2020-01-20	15.1948	15.0738	15.0738	6.3e-12
2020-01-20	15.0932	15.0738	15.0738	6.3e-12
2020-01-21	15.2255	15.2980	15.2980	6.6e-12
2020-01-21	15.2441	15.2980	15.2980	6.6e-12
2020-01-21	15.3685	15.2980	15.2980	6.6e-12
2020-01-21	15.3270	15.2980	15.2980	6.6e-12
2020-01-22	15.5394	15.5221	15.5221	7.0e-12
2020-01-22	15.4659	15.5221	15.5221	7.0e-12
2020-01-22	15.5676	15.5221	15.5221	7.0e-12
2020-01-22	15.5810	15.5221	15.5221	7.0e-12

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Plan cross-validation folds	1.16 ms	972 µs	1.2× slower
Cross-validate	5.76 ms	60.4 ms	10× faster
Score baselines	1.60 ms	1.55 ms	1.0× slower
Search settings	5.82 ms	62.3 ms	11× faster
Final holdout check	3.85 ms	23.5 ms	6.1× faster
Save report	266 µs	9 µs	30× slower
Load report	185 µs	5 µs	37× slower

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-01-17T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [],
    "events": [],
    "regressors": []
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 259200000,
    "cutoffs": [
      "2020-01-18T00:00:00.000Z",
      "2020-01-19T00:00:00.000Z"
    ]
  },
  "metricAggregation": "overall",
  "mapeZeroActual": "exclude",
  "candidates": [
    {
      "id": "control",
      "changepointPriorScale": 0.05
    }
  ],
  "holdoutRows": 8
}