Twelve folds and a search, sequential and parallel

match ← All logistic growth benchmarks

Twelve weekly cutoffs on the 256-day history, then three candidate settings compared on them. Python Prophet is also timed with parallel="threads" and parallel="processes", which must give the same forecasts and scores as running sequentially.

Input

benchmark/inputs/v1/logistic-implicit-floor-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 0.066 (0.079% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-06-21	80.2089	80.1755	80.2183	0.043
2020-06-22	79.9713	80.0772	80.1434	0.066
2020-06-23	80.2267	80.3008	80.3562	0.055
2020-06-24	80.8245	80.7672	80.7745	0.0072
2020-06-25	81.2461	81.2496	81.2485	0.0011
2020-06-26	81.3903	81.4796	81.5073	0.028
2020-06-27	81.3888	81.3562	81.3894	0.033
2020-06-28	81.1931	81.0855	81.1205	0.035
2020-06-29	80.9690	80.9785	81.0362	0.058
2020-06-30	81.1682	81.2007	81.2482	0.047
2020-07-01	81.7652	81.6698	81.6712	0.0014
2020-07-02	82.2132	82.1471	82.1407	0.0064
2020-07-03	82.2989	82.3717	82.3933	0.022
2020-07-04	82.2257	82.2503	82.2772	0.027
2020-07-05	82.0444	81.9785	82.0077	0.029
2020-07-06	81.8279	81.8657	81.9169	0.051
2020-07-07	81.9675	82.0844	82.1259	0.041
2020-07-08	82.5564	82.5566	82.5538	0.0028
2020-07-09	83.0729	83.0311	83.0208	0.010
2020-07-10	83.1751	83.2483	83.2652	0.017
2020-07-11	83.0764	83.1277	83.1498	0.022
2020-07-12	82.9440	82.8586	82.8826	0.024
2020-07-13	82.7931	82.7408	82.7860	0.045
2020-07-14	82.9070	82.9547	82.9908	0.036
2020-07-15	83.4626	83.4299	83.4235	0.0065
2020-07-16	84.0150	83.9045	83.8911	0.013
2020-07-17	84.1134	84.1162	84.1290	0.013
2020-07-18	83.9331	83.9954	84.0134	0.018
2020-07-19	83.7699	83.7277	83.7474	0.020
2020-07-20	83.6459	83.6081	83.6484	0.040
2020-07-21	83.7241	83.8170	83.8487	0.032
2020-07-22	84.2225	84.2941	84.2849	0.0092
2020-07-23	84.8099	84.7706	84.7547	0.016
2020-07-24	84.9611	84.9769	84.9865	0.0096
2020-07-25	84.7598	84.8537	84.8684	0.015
2020-07-26	84.5997	84.5897	84.6061	0.016
2020-07-27	84.5545	84.4696	84.5059	0.036
2020-07-28	84.6625	84.6729	84.7010	0.028
2020-07-29	85.1194	85.1501	85.1386	0.011
2020-07-30	85.7177	85.6297	85.6120	0.018
2020-07-31	85.9098	85.8330	85.8401	0.0071
2020-08-01	85.6622	85.7071	85.7192	0.012
2020-08-02	85.4318	85.4440	85.4578	0.014
2020-08-03	85.3968	85.3258	85.3588	0.033
2020-08-04	85.5084	85.5254	85.5506	0.025
2020-08-05	85.8967	86.0012	85.9883	0.013
2020-08-06	86.4713	86.4843	86.4653	0.019
2020-08-07	86.7202	86.6865	86.6916	0.0051
2020-08-08	86.4843	86.5563	86.5663	0.0099
2020-08-09	86.2238	86.2949	86.3065	0.012
2020-08-10	86.2360	86.1792	86.2096	0.030
2020-08-11	86.4223	86.3754	86.3982	0.023
2020-08-12	86.7988	86.8479	86.8339	0.014
2020-08-13	87.3514	87.3337	87.3139	0.020
2020-08-14	87.6488	87.5367	87.5402	0.0035
2020-08-15	87.4249	87.4032	87.4115	0.0083
2020-08-16	87.0921	87.1405	87.1505	0.010
2020-08-17	87.0732	87.0277	87.0560	0.028
2020-08-18	87.2845	87.2238	87.2448	0.021
2020-08-19	87.6215	87.6929	87.6780	0.015
2020-08-20	88.1043	88.1804	88.1600	0.020
2020-08-21	88.4210	88.3862	88.3885	0.0023
2020-08-22	88.2403	88.2492	88.2562	0.0070
2020-08-23	87.8794	87.9851	87.9937	0.0086
2020-08-24	87.8558	87.8752	87.9014	0.026
2020-08-25	88.1455	88.0714	88.0907	0.019
2020-08-26	88.5220	88.5362	88.5209	0.015
2020-08-27	88.9740	89.0235	89.0028	0.021
2020-08-28	89.3114	89.2321	89.2335	0.0014
2020-08-29	89.1865	89.0939	89.0998	0.0059
2020-08-30	88.7948	88.8267	88.8342	0.0075
2020-08-31	88.7072	88.7181	88.7428	0.025
2020-09-01	89.0078	88.9176	88.9356	0.018
2020-09-02	89.3886	89.3794	89.3638	0.016
2020-09-03	89.7661	89.8653	89.8445	0.021
2020-09-04	90.0652	90.0781	90.0788	0.00071
2020-09-05	89.9836	89.9392	89.9443	0.0051
2020-09-06	89.5946	89.6690	89.6756	0.0066
2020-09-07	89.4665	89.5610	89.5844	0.023
2020-09-08	89.8055	89.7631	89.7800	0.017
2020-09-09	90.2627	90.2225	90.2068	0.016
2020-09-10	90.6369	90.7053	90.6845	0.021
2020-09-11	90.9214	90.9203	90.9205	0.00018
2020-09-12	90.8993	90.7830	90.7874	0.0045

Timing

Median of all measured repetitions.

Effect Prophet runs cross-validation folds one at a time, so only Python Prophet has parallel timings.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Cross-validate	61.3 ms	170 ms	Not compared: different work
Cross-validate, Python parallel="threads"	—	164 ms	—
Cross-validate, Python parallel="processes"	—	1.49 s	—
Search settings	173 ms	528 ms	Not compared: different work
Search settings, Python parallel="threads"	—	500 ms	—
Search settings, Python parallel="processes"	—	4.55 s	—

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Cross-validate, Search settings: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "logistic",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-06-20T00:00:00.000Z",
      "2020-06-27T00:00:00.000Z",
      "2020-07-04T00:00:00.000Z",
      "2020-07-11T00:00:00.000Z",
      "2020-07-18T00:00:00.000Z",
      "2020-07-25T00:00:00.000Z",
      "2020-08-01T00:00:00.000Z",
      "2020-08-08T00:00:00.000Z",
      "2020-08-15T00:00:00.000Z",
      "2020-08-22T00:00:00.000Z",
      "2020-08-29T00:00:00.000Z",
      "2020-09-05T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "candidates": [
    {
      "id": "prior-low",
      "changepointPriorScale": 0.1
    },
    {
      "id": "prior-mid",
      "changepointPriorScale": 0.2
    },
    {
      "id": "prior-high",
      "changepointPriorScale": 0.5
    }
  ]
}