Twelve folds, sequential and parallel

match ← All flat growth benchmarks

Twelve weekly cutoffs on the 256-day history. Python Prophet is also timed with parallel="threads" and parallel="processes", which must give the same forecasts as running sequentially.

Input

benchmark/inputs/v1/flat-mixed-components-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 4.9e-7 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-06-21	28.9482	28.9148	28.9148	2.5e-7
2020-06-22	21.2484	21.3545	21.3545	5.1e-8
2020-06-23	23.9899	24.0677	24.0677	2.8e-9
2020-06-24	35.0514	35.0015	35.0015	2.5e-7
2020-06-25	47.1372	47.1401	47.1401	3.5e-7
2020-06-26	48.5686	48.6578	48.6578	4.9e-7
2020-06-27	41.1071	41.0755	41.0755	4.4e-7
2020-06-28	29.0247	28.9169	28.9169	1.1e-7
2020-06-29	34.9904	35.0005	35.0005	1.1e-7
2020-06-30	24.0292	24.0635	24.0635	2.2e-7
2020-07-01	36.2926	36.1993	36.1993	1.6e-7
2020-07-02	46.0073	45.9411	45.9411	6.1e-8
2020-07-03	48.5827	48.6546	48.6546	6.7e-10
2020-07-04	34.9774	35.0005	35.0005	1.1e-7
2020-07-05	28.9860	28.9211	28.9211	2.1e-8
2020-07-06	21.3124	21.3492	21.3492	4.6e-8
2020-07-07	25.1427	25.2630	25.2630	5.5e-8
2020-07-08	34.9999	35.0036	35.0036	1.6e-7
2020-07-09	35.0393	35.0007	35.0007	9.4e-8
2020-07-10	48.5786	48.6504	48.6504	2.0e-7
2020-07-11	41.0239	41.0764	41.0764	9.0e-8
2020-07-12	29.0086	28.9247	28.9247	5.8e-8
2020-07-13	22.6022	22.5471	22.5471	4.6e-8
2020-07-14	34.9539	35.0004	35.0004	1.1e-7
2020-07-15	35.0337	35.0041	35.0041	1.5e-7
2020-07-16	46.0561	45.9451	45.9451	1.8e-7
2020-07-17	48.6472	48.6477	48.6477	1.6e-7
2020-07-18	41.0122	41.0756	41.0756	1.1e-7
2020-07-19	36.2416	36.2004	36.2004	1.8e-7
2020-07-20	21.3889	21.3513	21.3513	9.1e-8
2020-07-21	23.9604	24.0555	24.0555	3.3e-8
2020-07-22	34.9296	35.0053	35.0053	2.0e-7
2020-07-23	45.9880	45.9491	45.9491	4.3e-7
2020-07-24	34.9840	35.0011	35.0011	2.0e-7
2020-07-25	42.1779	42.2721	42.2721	3.0e-7
2020-07-26	28.9370	28.9275	28.9275	7.3e-8
2020-07-27	21.4383	21.3533	21.3533	1.4e-7
2020-07-28	24.0404	24.0521	24.0521	2.8e-7
2020-07-29	34.9689	35.0005	35.0005	2.3e-7
2020-07-30	46.0389	45.9514	45.9514	3.8e-7
2020-07-31	49.9255	49.8456	49.8456	2.5e-7
2020-08-01	41.0247	41.0694	41.0694	1.3e-7
2020-08-02	28.9142	28.9273	28.9273	5.0e-8
2020-08-03	35.0753	35.0010	35.0010	1.8e-7
2020-08-04	24.0327	24.0514	24.0514	2.5e-7
2020-08-05	34.8932	35.0010	35.0010	3.1e-7
2020-08-06	47.1400	47.1548	47.1548	2.7e-7
2020-08-07	48.6840	48.6493	48.6493	1.6e-7
2020-08-08	34.9210	35.0010	35.0010	1.8e-7
2020-08-09	28.8553	28.9266	28.9266	5.5e-9
2020-08-10	21.4151	21.3565	21.3565	1.1e-7
2020-08-11	24.0965	24.0501	24.0501	1.2e-7
2020-08-12	36.1457	36.1976	36.1976	7.1e-9
2020-08-13	35.0253	35.0004	35.0004	3.1e-8
2020-08-14	48.7637	48.6511	48.6511	1.9e-7
2020-08-15	41.0876	41.0671	41.0671	1.4e-7
2020-08-16	28.8756	28.9246	28.9246	3.3e-8
2020-08-17	21.4046	21.3588	21.3588	9.7e-8
2020-08-18	36.2570	36.1993	36.1993	2.2e-8
2020-08-19	34.9214	34.9954	34.9954	3.7e-8
2020-08-20	45.8771	45.9542	45.9542	1.1e-7
2020-08-21	48.6894	48.6558	48.6558	2.0e-7
2020-08-22	41.0568	41.0680	41.0680	1.6e-7
2020-08-23	34.8913	35.0006	35.0006	1.4e-7
2020-08-24	22.5415	22.5602	22.5602	1.1e-7
2020-08-25	24.1269	24.0532	24.0532	2.6e-8
2020-08-26	34.9766	34.9920	34.9920	2.1e-8
2020-08-27	45.9018	45.9518	45.9518	1.1e-7
2020-08-28	35.0860	35.0006	35.0006	1.4e-7
2020-08-29	41.1585	41.0676	41.0676	2.8e-7
2020-08-30	30.0879	30.1225	30.1225	2.0e-7
2020-08-31	21.3487	21.3596	21.3596	2.9e-7
2020-09-01	24.1452	24.0553	24.0553	2.6e-7
2020-09-02	34.9994	35.0010	35.0010	1.4e-7
2020-09-03	45.8503	45.9492	45.9492	2.4e-8
2020-09-04	48.6453	48.6573	48.6573	1.2e-8
2020-09-05	42.3122	42.2698	42.2698	6.3e-8
2020-09-06	28.8445	28.9217	28.9217	2.2e-7
2020-09-07	34.9139	35.0008	35.0008	5.1e-8
2020-09-08	24.0999	24.0584	24.0584	2.8e-8
2020-09-09	35.0306	34.9921	34.9921	1.2e-7
2020-09-10	45.8784	45.9457	45.9457	8.4e-8
2020-09-11	49.8590	49.8567	49.8567	4.7e-8
2020-09-12	35.1111	35.0008	35.0008	5.1e-8

Timing

Median of all measured repetitions.

Effect Prophet runs cross-validation folds one at a time, so only Python Prophet has parallel timings.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Cross-validate	27.6 ms	184 ms	Not compared: different work
Cross-validate, Python parallel="threads"	—	199 ms	—
Cross-validate, Python parallel="processes"	—	1.50 s	—

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Cross-validate: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-06-20T00:00:00.000Z",
      "2020-06-27T00:00:00.000Z",
      "2020-07-04T00:00:00.000Z",
      "2020-07-11T00:00:00.000Z",
      "2020-07-18T00:00:00.000Z",
      "2020-07-25T00:00:00.000Z",
      "2020-08-01T00:00:00.000Z",
      "2020-08-08T00:00:00.000Z",
      "2020-08-15T00:00:00.000Z",
      "2020-08-22T00:00:00.000Z",
      "2020-08-29T00:00:00.000Z",
      "2020-09-05T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}