Choosing settings and a final holdout

match ← All linear growth benchmarks

Two candidate settings compared on the same folds, then a one-time holdout check.

Input

benchmark/inputs/v1/linear-offset-scaling.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 2.9e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-11	107.2568	107.1688	107.1688	2.9e-11
2020-03-12	107.4776	107.4962	107.4962	2.9e-11
2020-03-13	107.6213	107.7018	107.7018	2.8e-11
2020-03-14	107.7396	107.6554	107.6554	2.8e-11
2020-03-15	107.6534	107.5295	107.5295	2.7e-11
2020-03-16	107.5228	107.5532	107.5532	2.8e-11
2020-03-17	107.7277	107.7317	107.7317	2.8e-11
2020-03-18	108.1448	108.0262	108.0262	1.8e-11
2020-03-19	108.3767	108.3442	108.3442	1.8e-11
2020-03-20	108.4456	108.5444	108.5444	1.8e-11
2020-03-21	108.5155	108.5134	108.5134	1.7e-11
2020-03-22	108.4600	108.3912	108.3912	1.7e-11
2020-03-23	108.3204	108.4014	108.4014	1.8e-11
2020-03-24	108.4700	108.5823	108.5823	1.8e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Search settings	20.5 ms	164 ms	8.0× faster
Final holdout check	6.35 ms	29.7 ms	4.7× faster
Save report	234 µs	9 µs	26× slower
Load report	234 µs	4 µs	59× slower

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "overall",
  "mapeZeroActual": "exclude",
  "candidates": [
    {
      "id": "prior-low",
      "changepointPriorScale": 0.05
    },
    {
      "id": "prior-high",
      "changepointPriorScale": 0.2
    }
  ],
  "holdoutRows": 8
}