Choosing settings by a custom metric

match ← All linear growth benchmarks

Two candidate settings ranked by a custom WAPE metric, next to the same search by MAE.

Input

benchmark/inputs/v1/linear-offset-scaling.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 2.9e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-11	107.2568	107.1688	107.1688	2.9e-11
2020-03-12	107.4776	107.4962	107.4962	2.9e-11
2020-03-13	107.6213	107.7018	107.7018	2.8e-11
2020-03-14	107.7396	107.6554	107.6554	2.8e-11
2020-03-15	107.6534	107.5295	107.5295	2.7e-11
2020-03-16	107.5228	107.5532	107.5532	2.8e-11
2020-03-17	107.7277	107.7317	107.7317	2.8e-11
2020-03-18	108.1448	108.0262	108.0262	1.8e-11
2020-03-19	108.3767	108.3442	108.3442	1.8e-11
2020-03-20	108.4456	108.5444	108.5444	1.8e-11
2020-03-21	108.5155	108.5134	108.5134	1.7e-11
2020-03-22	108.4600	108.3912	108.3912	1.7e-11
2020-03-23	108.3204	108.4014	108.4014	1.8e-11
2020-03-24	108.4700	108.5823	108.5823	1.8e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Search settings	20.4 ms	143 ms	Not compared: different work
Search settings by a custom metric	17.7 ms	151 ms	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Search settings, Search settings by a custom metric: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "overall",
  "candidates": [
    {
      "id": "prior-low",
      "changepointPriorScale": 0.05
    },
    {
      "id": "prior-high",
      "changepointPriorScale": 0.2
    }
  ]
}