Monthly, custom and scaled metrics

match ← All logistic growth benchmarks

Three 91-day folds scored by calendar month, with custom metrics and with MASE, one score per horizon. The scores Effect gives Python's own forecasts must equal Python's exactly. Python Prophet has no MASE, so Python registers its own; each fold's scale must match Effect's.

Input

benchmark/inputs/v1/logistic-implicit-floor-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 0.17 (0.20% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-05-02	72.6035	72.6122	72.7469	0.13
2020-05-03	72.4123	72.4474	72.5852	0.14
2020-05-04	72.4830	72.4226	72.5908	0.17
2020-05-05	72.7222	72.6934	72.8464	0.15
2020-05-06	73.1532	73.2454	73.3305	0.085
2020-05-07	73.7654	73.8176	73.8886	0.071
2020-05-08	74.1287	74.0903	74.2027	0.11
2020-05-09	73.9798	74.0177	74.1412	0.12
2020-05-10	73.7317	73.8302	73.9562	0.13
2020-05-11	73.8025	73.7835	73.9394	0.16
2020-05-12	74.1035	74.0333	74.1736	0.14
2020-05-13	74.5278	74.5651	74.6371	0.072
2020-05-14	75.0906	75.1178	75.1753	0.058
2020-05-15	75.4742	75.3718	75.4704	0.099
2020-05-16	75.3460	75.2812	75.3907	0.11
2020-05-17	75.0252	75.0764	75.1882	0.11
2020-05-18	75.0296	75.0131	75.1547	0.14
2020-05-19	75.3375	75.2469	75.3728	0.13
2020-05-20	75.7290	75.7634	75.8208	0.057
2020-05-21	76.1988	76.3014	76.3443	0.043
2020-05-22	76.5589	76.5412	76.6252	0.084
2020-05-23	76.4649	76.4371	76.5320	0.095
2020-05-24	76.1154	76.2192	76.3164	0.097
2020-05-25	76.0805	76.1434	76.2704	0.13
2020-05-26	76.4393	76.3652	76.4766	0.11
2020-05-27	76.8810	76.8702	76.9132	0.043
2020-05-28	77.3189	77.3971	77.4257	0.029
2020-05-29	77.6713	77.6263	77.6961	0.070
2020-05-30	77.6312	77.5120	77.5928	0.081
2020-05-31	77.2715	77.2844	77.3677	0.083
2020-06-01	77.1609	77.1992	77.3124	0.11
2020-06-02	77.5051	77.4120	77.5097	0.098
2020-06-03	77.9590	77.9083	77.9379	0.030
2020-06-04	78.3281	78.4270	78.4424	0.015
2020-06-05	78.6098	78.6483	78.7050	0.057
2020-06-06	78.5899	78.5264	78.5943	0.068
2020-06-07	78.2419	78.2914	78.3621	0.071
2020-06-08	78.0790	78.1992	78.3001	0.10
2020-06-09	78.4263	78.4053	78.4909	0.086
2020-06-10	78.9532	78.8952	78.9129	0.018
2020-06-11	79.3352	79.4077	79.4114	0.0037
2020-06-12	79.5870	79.6231	79.6684	0.045
2020-06-13	79.6086	79.4955	79.5523	0.057
2020-06-14	79.3094	79.2551	79.3149	0.060
2020-06-15	79.1014	79.1578	79.2479	0.090
2020-06-16	79.4016	79.3589	79.4339	0.075
2020-06-17	79.9515	79.8440	79.8514	0.0074
2020-06-18	80.3159	80.3519	80.3455	0.0063
2020-06-19	80.4808	80.5629	80.5984	0.035
2020-06-20	80.4760	80.4311	80.4783	0.047
2020-06-21	80.2089	80.1867	80.2371	0.050
2020-06-22	79.9713	80.0854	80.1664	0.081
2020-06-23	80.2267	80.2829	80.3490	0.066
2020-06-24	80.8245	80.7644	80.7631	0.0013
2020-06-25	81.2461	81.2689	81.2541	0.015
2020-06-26	81.3903	81.4767	81.5039	0.027
2020-06-27	81.3888	81.3417	81.3809	0.039
2020-06-28	81.1931	81.0943	81.1369	0.043
2020-06-29	80.9690	80.9902	81.0637	0.073
2020-06-30	81.1682	81.1849	81.2437	0.059
2020-07-01	81.7652	81.6639	81.6554	0.0085
2020-07-02	82.2132	82.1658	82.1441	0.022
2020-07-03	82.2989	82.3712	82.3916	0.020
2020-07-04	82.2257	82.2339	82.2665	0.033
2020-07-05	82.0444	81.9843	82.0205	0.036
2020-07-06	81.8279	81.8781	81.9453	0.067
2020-07-07	81.9675	82.0708	82.1235	0.053
2020-07-08	82.5564	82.5478	82.5334	0.014
2020-07-09	83.0729	83.0479	83.0204	0.027
2020-07-10	83.1751	83.2514	83.2663	0.015
2020-07-11	83.0764	83.1125	83.1397	0.027
2020-07-12	82.9440	82.8612	82.8922	0.031
2020-07-13	82.7931	82.7535	82.8156	0.062
2020-07-14	82.9070	82.9446	82.9924	0.048
2020-07-15	83.4626	83.4202	83.4011	0.019
2020-07-16	84.0150	83.9189	83.8868	0.032
2020-07-17	84.1134	84.1211	84.1316	0.010
2020-07-18	83.9331	83.9809	84.0038	0.023
2020-07-19	83.7699	83.7285	83.7552	0.027
2020-07-20	83.6459	83.6195	83.6776	0.058
2020-07-21	83.7241	83.8096	83.8535	0.044
2020-07-22	84.2225	84.2841	84.2612	0.023
2020-07-23	84.8099	84.7818	84.7460	0.036
2020-07-24	84.9611	84.9830	84.9899	0.0069
2020-07-25	84.7598	84.8419	84.8613	0.019
2020-07-26	84.5997	84.5886	84.6120	0.023
2020-07-27	84.5545	84.4788	84.5336	0.055
2020-07-28	84.6625	84.6680	84.7088	0.041
2020-07-29	85.1194	85.1417	85.1158	0.026
2020-07-30	85.7177	85.6387	85.6000	0.039
2020-07-31	85.9098	85.8392	85.8433	0.0041
2020-05-30	77.6312	77.5152	77.5894	0.074
2020-05-31	77.2715	77.2711	77.3467	0.076
2020-06-01	77.1609	77.2013	77.3019	0.10
2020-06-02	77.5051	77.4255	77.5135	0.088
2020-06-03	77.9590	77.9015	77.9341	0.033
2020-06-04	78.3281	78.4164	78.4380	0.022
2020-06-05	78.6098	78.6577	78.7126	0.055
2020-06-06	78.5899	78.5296	78.5920	0.062
2020-06-07	78.2419	78.2782	78.3422	0.064
2020-06-08	78.0790	78.2014	78.2906	0.089
2020-06-09	78.4263	78.4189	78.4957	0.077
2020-06-10	78.9532	78.8885	78.9100	0.022
2020-06-11	79.3352	79.3972	79.4080	0.011
2020-06-12	79.5870	79.6325	79.6769	0.044
2020-06-13	79.6086	79.4989	79.5508	0.052
2020-06-14	79.3094	79.2420	79.2957	0.054
2020-06-15	79.1014	79.1600	79.2391	0.079
2020-06-16	79.4016	79.3725	79.4395	0.067
2020-06-17	79.9515	79.8373	79.8493	0.012
2020-06-18	80.3159	80.3414	80.3429	0.0015
2020-06-19	80.4808	80.5724	80.6076	0.035
2020-06-20	80.4760	80.4345	80.4775	0.043
2020-06-21	80.2089	80.1735	80.2186	0.045
2020-06-22	79.9713	80.0877	80.1583	0.071
2020-06-23	80.2267	80.2965	80.3552	0.059
2020-06-24	80.8245	80.7578	80.7616	0.0038
2020-06-25	81.2461	81.2585	81.2520	0.0064
2020-06-26	81.3903	81.4862	81.5137	0.027
2020-06-27	81.3888	81.3452	81.3807	0.036
2020-06-28	81.1931	81.0813	81.1190	0.038
2020-06-29	80.9690	80.9926	81.0561	0.063
2020-06-30	81.1682	81.1986	81.2503	0.052
2020-07-01	81.7652	81.6573	81.6543	0.0029
2020-07-02	82.2132	82.1554	82.1424	0.013
2020-07-03	82.2989	82.3807	82.4018	0.021
2020-07-04	82.2257	82.2374	82.2667	0.029
2020-07-05	82.0444	81.9713	82.0030	0.032
2020-07-06	81.8279	81.8805	81.9381	0.058
2020-07-07	81.9675	82.0845	82.1305	0.046
2020-07-08	82.5564	82.5412	82.5327	0.0085
2020-07-09	83.0729	83.0375	83.0191	0.018
2020-07-10	83.1751	83.2610	83.2769	0.016
2020-07-11	83.0764	83.1160	83.1402	0.024
2020-07-12	82.9440	82.8482	82.8750	0.027
2020-07-13	82.7931	82.7559	82.8087	0.053
2020-07-14	82.9070	82.9584	82.9998	0.041
2020-07-15	83.4626	83.4136	83.4007	0.013
2020-07-16	84.0150	83.9085	83.8858	0.023
2020-07-17	84.1134	84.1308	84.1424	0.012
2020-07-18	83.9331	83.9845	84.0046	0.020
2020-07-19	83.7699	83.7155	83.7383	0.023
2020-07-20	83.6459	83.6220	83.6710	0.049
2020-07-21	83.7241	83.8234	83.8610	0.038
2020-07-22	84.2225	84.2776	84.2610	0.017
2020-07-23	84.8099	84.7714	84.7452	0.026
2020-07-24	84.9611	84.9927	85.0010	0.0083
2020-07-25	84.7598	84.8455	84.8623	0.017
2020-07-26	84.5997	84.5756	84.5952	0.020
2020-07-27	84.5545	84.4812	84.5271	0.046
2020-07-28	84.6625	84.6818	84.7165	0.035
2020-07-29	85.1194	85.1352	85.1158	0.019
2020-07-30	85.7177	85.6283	85.5994	0.029
2020-07-31	85.9098	85.8489	85.8545	0.0056
2020-08-01	85.6622	85.7010	85.7152	0.014
2020-08-02	85.4318	85.4304	85.4475	0.017
2020-08-03	85.3968	85.3354	85.3789	0.044
2020-08-04	85.5084	85.5354	85.5677	0.032
2020-08-05	85.8967	85.9882	85.9665	0.022
2020-08-06	86.4713	86.4808	86.4496	0.031
2020-08-07	86.7202	86.7008	86.7043	0.0035
2020-08-08	86.4843	86.5524	86.5646	0.012
2020-08-09	86.2238	86.2814	86.2965	0.015
2020-08-10	86.2360	86.1859	86.2275	0.042
2020-08-11	86.4223	86.3854	86.4159	0.030
2020-08-12	86.7988	86.8378	86.8143	0.023
2020-08-13	87.3514	87.3300	87.2971	0.033
2020-08-14	87.6488	87.5496	87.5514	0.0018
2020-08-15	87.4249	87.4008	87.4114	0.011
2020-08-16	87.0921	87.1294	87.1430	0.014
2020-08-17	87.0732	87.0336	87.0737	0.040
2020-08-18	87.2845	87.2328	87.2619	0.029
2020-08-19	87.6215	87.6849	87.6600	0.025
2020-08-20	88.1043	88.1768	88.1425	0.034
2020-08-21	88.4210	88.3961	88.3966	0.00048
2020-08-22	88.2403	88.2470	88.2564	0.0093
2020-08-23	87.8794	87.9754	87.9877	0.012
2020-08-24	87.8558	87.8793	87.9182	0.039
2020-08-25	88.1455	88.0783	88.1062	0.028
2020-08-26	88.5220	88.5301	88.5041	0.026
2020-08-27	88.9740	89.0218	88.9865	0.035
2020-08-28	89.3114	89.2409	89.2403	0.00054
2020-06-13	79.6086	79.5063	79.5574	0.051
2020-06-14	79.3094	79.2413	79.2938	0.052
2020-06-15	79.1014	79.1530	79.2289	0.076
2020-06-16	79.4016	79.3767	79.4415	0.065
2020-06-17	79.9515	79.8430	79.8573	0.014
2020-06-18	80.3159	80.3350	80.3399	0.0049
2020-06-19	80.4808	80.5695	80.6051	0.036
2020-06-20	80.4760	80.4419	80.4841	0.042
2020-06-21	80.2089	80.1729	80.2167	0.044
2020-06-22	79.9713	80.0807	80.1482	0.067
2020-06-23	80.2267	80.3007	80.3572	0.057
2020-06-24	80.8245	80.7635	80.7697	0.0062
2020-06-25	81.2461	81.2521	81.2491	0.0030
2020-06-26	81.3903	81.4833	81.5113	0.028
2020-06-27	81.3888	81.3526	81.3873	0.035
2020-06-28	81.1931	81.0806	81.1171	0.036
2020-06-29	80.9690	80.9856	81.0459	0.060
2020-06-30	81.1682	81.2028	81.2524	0.050
2020-07-01	81.7652	81.6630	81.6624	0.00051
2020-07-02	82.2132	82.1490	82.1395	0.0095
2020-07-03	82.2989	82.3778	82.3995	0.022
2020-07-04	82.2257	82.2448	82.2733	0.029
2020-07-05	82.0444	81.9706	82.0011	0.030
2020-07-06	81.8279	81.8735	81.9280	0.055
2020-07-07	81.9675	82.0887	82.1326	0.044
2020-07-08	82.5564	82.5469	82.5408	0.0060
2020-07-09	83.0729	83.0311	83.0162	0.015
2020-07-10	83.1751	83.2581	83.2745	0.016
2020-07-11	83.0764	83.1234	83.1469	0.024
2020-07-12	82.9440	82.8476	82.8731	0.026
2020-07-13	82.7931	82.7488	82.7986	0.050
2020-07-14	82.9070	82.9625	83.0019	0.039
2020-07-15	83.4626	83.4193	83.4088	0.010
2020-07-16	84.0150	83.9021	83.8829	0.019
2020-07-17	84.1134	84.1278	84.1401	0.012
2020-07-18	83.9331	83.9918	84.0112	0.019
2020-07-19	83.7699	83.7148	83.7365	0.022
2020-07-20	83.6459	83.6149	83.6609	0.046
2020-07-21	83.7241	83.8275	83.8632	0.036
2020-07-22	84.2225	84.2832	84.2691	0.014
2020-07-23	84.8099	84.7650	84.7424	0.023
2020-07-24	84.9611	84.9898	84.9986	0.0089
2020-07-25	84.7598	84.8528	84.8690	0.016
2020-07-26	84.5997	84.5749	84.5934	0.018
2020-07-27	84.5545	84.4742	84.5171	0.043
2020-07-28	84.6625	84.6860	84.7186	0.033
2020-07-29	85.1194	85.1409	85.1239	0.017
2020-07-30	85.7177	85.6219	85.5965	0.025
2020-07-31	85.9098	85.8459	85.8521	0.0062
2020-08-01	85.6622	85.7083	85.7219	0.014
2020-08-02	85.4318	85.4298	85.4457	0.016
2020-08-03	85.3968	85.3284	85.3688	0.040
2020-08-04	85.5084	85.5396	85.5699	0.030
2020-08-05	85.8967	85.9939	85.9747	0.019
2020-08-06	86.4713	86.4744	86.4468	0.028
2020-08-07	86.7202	86.6979	86.7019	0.0041
2020-08-08	86.4843	86.5597	86.5713	0.012
2020-08-09	86.2238	86.2807	86.2947	0.014
2020-08-10	86.2360	86.1788	86.2174	0.039
2020-08-11	86.4223	86.3896	86.4181	0.028
2020-08-12	86.7988	86.8435	86.8225	0.021
2020-08-13	87.3514	87.3236	87.2942	0.029
2020-08-14	87.6488	87.5467	87.5491	0.0024
2020-08-15	87.4249	87.4082	87.4181	0.0099
2020-08-16	87.0921	87.1288	87.1412	0.012
2020-08-17	87.0732	87.0266	87.0636	0.037
2020-08-18	87.2845	87.2370	87.2640	0.027
2020-08-19	87.6215	87.6906	87.6682	0.022
2020-08-20	88.1043	88.1703	88.1397	0.031
2020-08-21	88.4210	88.3931	88.3943	0.0011
2020-08-22	88.2403	88.2544	88.2630	0.0087
2020-08-23	87.8794	87.9747	87.9859	0.011
2020-08-24	87.8558	87.8723	87.9081	0.036
2020-08-25	88.1455	88.0824	88.1083	0.026
2020-08-26	88.5220	88.5358	88.5123	0.023
2020-08-27	88.9740	89.0153	88.9836	0.032
2020-08-28	89.3114	89.2379	89.2380	0.00011
2020-08-29	89.1865	89.0989	89.1066	0.0077
2020-08-30	88.7948	88.8191	88.8294	0.010
2020-08-31	88.7072	88.7165	88.7514	0.035
2020-09-01	89.0078	88.9264	88.9515	0.025
2020-09-02	89.3886	89.3796	89.3553	0.024
2020-09-03	89.7661	89.8590	89.8265	0.033
2020-09-04	90.0652	90.0815	90.0808	0.00068
2020-09-05	89.9836	89.9423	89.9493	0.0069
2020-09-06	89.5946	89.6623	89.6719	0.0096
2020-09-07	89.4665	89.5595	89.5938	0.034
2020-09-08	89.8055	89.7694	89.7938	0.024
2020-09-09	90.2627	90.2225	90.1975	0.025
2020-09-10	90.6369	90.7017	90.6685	0.033
2020-09-11	90.9214	90.9241	90.9228	0.0013

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Compute metrics	467 µs	5.58 ms	Not compared: different work
Compute metrics by calendar month	359 µs	5.04 ms	Not compared: different work
Compute custom metrics	1.26 ms	12.9 ms	Not compared: different work
Compute MASE	1.62 ms	5.38 ms	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Compute metrics, Compute metrics by calendar month, Compute custom metrics, Compute MASE: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "logistic",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 7862400000,
    "cutoffs": [
      "2020-05-01T00:00:00.000Z",
      "2020-05-29T00:00:00.000Z",
      "2020-06-12T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}