Cross-validation with ranges, eight months

match ← All flat growth benchmarks

The same long folds with simulated futures for interval coverage.

Input

benchmark/inputs/v1/flat-mixed-components-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 2.4e-7 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-08-13	35.0253	35.0005	35.0005	1.6e-7
2020-08-14	48.7637	48.6525	48.6525	1.8e-7
2020-08-15	41.0876	41.0662	41.0662	2.4e-7
2020-08-16	28.8756	28.9251	28.9251	2.4e-7
2020-08-17	21.4046	21.3585	21.3585	1.8e-7
2020-08-18	36.2570	36.1993	36.1993	1.6e-7
2020-08-19	34.9214	34.9960	34.9960	7.3e-8
2020-08-20	45.8771	45.9534	45.9534	4.3e-8
2020-08-21	48.6894	48.6567	48.6567	9.5e-9
2020-08-22	41.0568	41.0671	41.0671	9.2e-8
2020-08-23	34.8913	35.0009	35.0009	4.9e-8
2020-08-24	22.5415	22.5599	22.5599	1.2e-7
2020-08-25	24.1269	24.0527	24.0527	4.7e-8
2020-08-26	34.9766	34.9930	34.9930	2.7e-8

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Cross-validate with ranges	8.02 ms	46.5 ms	5.8× faster

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-08-12T00:00:00.000Z",
      "2020-08-19T00:00:00.000Z"
    ]
  },
  "metricAggregation": "rolling",
  "mapeZeroActual": "exclude",
  "interval": {
    "seed": 19,
    "samples": 128,
    "intervalWidth": 0.8
  }
}