Search with a failing candidate

match ← All linear growth benchmarks

Three candidates, one of which is designed to fail. The search records it and continues.

Input

benchmark/inputs/v1/linear-offset-scaling.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 2.9e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-11	107.2568	107.1688	107.1688	2.9e-11
2020-03-12	107.4776	107.4962	107.4962	2.9e-11
2020-03-13	107.6213	107.7018	107.7018	2.8e-11
2020-03-14	107.7396	107.6554	107.6554	2.8e-11
2020-03-15	107.6534	107.5295	107.5295	2.7e-11
2020-03-16	107.5228	107.5532	107.5532	2.8e-11
2020-03-17	107.7277	107.7317	107.7317	2.8e-11
2020-03-18	108.1448	108.0262	108.0262	1.8e-11
2020-03-19	108.3767	108.3442	108.3442	1.8e-11
2020-03-20	108.4456	108.5444	108.5444	1.8e-11
2020-03-21	108.5155	108.5134	108.5134	1.7e-11
2020-03-22	108.4600	108.3912	108.3912	1.7e-11
2020-03-23	108.3204	108.4014	108.4014	1.8e-11
2020-03-24	108.4700	108.5823	108.5823	1.8e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Search settings	21.4 ms	153 ms	Not compared: different work
Final holdout check	6.01 ms	30.1 ms	Not compared: different work
Save report	286 µs	8 µs	Not compared: different work
Load report	210 µs	5 µs	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Search settings, Final holdout check, Save report, Load report: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "candidates": [
    {
      "id": "prior-low",
      "changepointPriorScale": 0.05
    },
    {
      "id": "prior-high",
      "changepointPriorScale": 0.2
    },
    {
      "id": "missing-capacity",
      "changepointPriorScale": 0.1,
      "growth": "logistic"
    }
  ],
  "expectedFailures": 1,
  "holdoutRows": 8
}