Monthly, custom and scaled metrics

match ← All flat growth benchmarks

Three 91-day folds scored by calendar month, with custom metrics and with MASE, in rolling windows. The scores Effect gives Python's own forecasts must equal Python's exactly. Python Prophet has no MASE, so Python registers its own; each fold's scale must match Effect's.

Input

benchmark/inputs/v1/flat-mixed-components-large.json: 256 history rows and 64 forecast rows.

Output

Largest difference: 4.1e-7 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-05-02	42.2518	42.2628	42.2628	1.5e-7
2020-05-03	28.8913	28.9279	28.9279	7.7e-8
2020-05-04	21.4235	21.3616	21.3616	1.5e-7
2020-05-05	35.0208	35.0022	35.0022	2.4e-7
2020-05-06	34.8995	34.9953	34.9953	3.9e-7
2020-05-07	45.9081	45.9608	45.9608	3.8e-7
2020-05-08	49.8937	49.8545	49.8545	2.8e-7
2020-05-09	41.0225	41.0648	41.0648	1.3e-7
2020-05-10	34.9022	35.0022	35.0022	2.4e-7
2020-05-11	21.3819	21.3616	21.3616	1.5e-7
2020-05-12	24.1165	24.0484	24.0484	2.9e-7
2020-05-13	34.9543	34.9953	34.9953	3.9e-7
2020-05-14	47.1330	47.1587	47.1587	4.0e-7
2020-05-15	35.1086	35.0022	35.0022	2.4e-7
2020-05-16	41.1251	41.0648	41.0648	1.3e-7
2020-05-17	28.8750	28.9279	28.9279	7.7e-8
2020-05-18	21.3793	21.3616	21.3616	1.5e-7
2020-05-19	24.1368	24.0484	24.0484	2.9e-7
2020-05-20	36.1571	36.2001	36.2001	2.6e-7
2020-05-21	45.8576	45.9608	45.9608	3.8e-7
2020-05-22	48.6729	48.6566	48.6566	2.6e-7
2020-05-23	41.0880	41.0648	41.0648	1.3e-7
2020-05-24	28.8224	28.9279	28.9279	7.7e-8
2020-05-25	34.9489	35.0022	35.0022	2.4e-7
2020-05-26	25.3202	25.2464	25.2464	3.1e-7
2020-05-27	35.0023	34.9953	34.9953	3.9e-7
2020-05-28	45.8819	45.9608	45.9608	3.8e-7
2020-05-29	48.7001	48.6566	48.6566	2.6e-7
2020-05-30	35.1050	35.0022	35.0022	2.4e-7
2020-05-31	28.9133	28.9279	28.9279	7.7e-8
2020-06-01	22.5244	22.5596	22.5596	1.7e-7
2020-06-02	24.1392	24.0484	24.0484	2.9e-7
2020-06-03	35.0420	34.9953	34.9953	3.9e-7
2020-06-04	34.9155	35.0022	35.0022	2.4e-7
2020-06-05	48.6166	48.6566	48.6566	2.6e-7
2020-06-06	41.1237	41.0648	41.0648	1.3e-7
2020-06-07	30.0765	30.1258	30.1258	9.7e-8
2020-06-08	21.2425	21.3616	21.3616	1.5e-7
2020-06-09	35.0127	35.0022	35.0022	2.4e-7
2020-06-10	35.0493	34.9953	34.9953	3.9e-7
2020-06-11	45.8875	45.9608	45.9608	3.8e-7
2020-06-12	48.6189	48.6566	48.6566	2.6e-7
2020-06-13	42.3732	42.2628	42.2628	1.5e-7
2020-06-14	35.0547	35.0022	35.0022	2.4e-7
2020-06-15	21.3063	21.3616	21.3616	1.5e-7
2020-06-16	24.0887	24.0484	24.0484	2.9e-7
2020-06-17	35.0988	34.9953	34.9953	3.9e-7
2020-06-18	45.9240	45.9608	45.9608	3.8e-7
2020-06-19	36.1239	36.2001	36.2001	2.6e-7
2020-06-20	41.1050	41.0648	41.0648	1.3e-7
2020-06-21	28.9482	28.9279	28.9279	7.7e-8
2020-06-22	21.2484	21.3616	21.3616	1.5e-7
2020-06-23	23.9899	24.0484	24.0484	2.9e-7
2020-06-24	35.0514	35.0022	35.0022	2.4e-7
2020-06-25	47.1372	47.1587	47.1587	4.0e-7
2020-06-26	48.5686	48.6566	48.6566	2.6e-7
2020-06-27	41.1071	41.0648	41.0648	1.3e-7
2020-06-28	29.0247	28.9279	28.9279	7.7e-8
2020-06-29	34.9904	35.0022	35.0022	2.4e-7
2020-06-30	24.0292	24.0484	24.0484	2.9e-7
2020-07-01	36.2926	36.1932	36.1932	4.1e-7
2020-07-02	46.0073	45.9608	45.9608	3.8e-7
2020-07-03	48.5827	48.6566	48.6566	2.6e-7
2020-07-04	34.9774	35.0022	35.0022	2.4e-7
2020-07-05	28.9860	28.9279	28.9279	7.7e-8
2020-07-06	21.3124	21.3616	21.3616	1.5e-7
2020-07-07	25.1427	25.2464	25.2464	3.1e-7
2020-07-08	34.9999	34.9953	34.9953	3.9e-7
2020-07-09	35.0393	35.0022	35.0022	2.4e-7
2020-07-10	48.5786	48.6566	48.6566	2.6e-7
2020-07-11	41.0239	41.0648	41.0648	1.3e-7
2020-07-12	29.0086	28.9279	28.9279	7.7e-8
2020-07-13	22.6022	22.5596	22.5596	1.7e-7
2020-07-14	34.9539	35.0022	35.0022	2.4e-7
2020-07-15	35.0337	34.9953	34.9953	3.9e-7
2020-07-16	46.0561	45.9608	45.9608	3.8e-7
2020-07-17	48.6472	48.6566	48.6566	2.6e-7
2020-07-18	41.0122	41.0648	41.0648	1.3e-7
2020-07-19	36.2416	36.2001	36.2001	2.6e-7
2020-07-20	21.3889	21.3616	21.3616	1.5e-7
2020-07-21	23.9604	24.0484	24.0484	2.9e-7
2020-07-22	34.9296	34.9953	34.9953	3.9e-7
2020-07-23	45.9880	45.9608	45.9608	3.8e-7
2020-07-24	34.9840	35.0022	35.0022	2.4e-7
2020-07-25	42.1779	42.2628	42.2628	1.5e-7
2020-07-26	28.9370	28.9279	28.9279	7.7e-8
2020-07-27	21.4383	21.3616	21.3616	1.5e-7
2020-07-28	24.0404	24.0484	24.0484	2.9e-7
2020-07-29	34.9689	35.0022	35.0022	2.4e-7
2020-07-30	46.0389	45.9608	45.9608	3.8e-7
2020-07-31	49.9255	49.8545	49.8545	2.8e-7
2020-05-30	35.1050	35.0004	35.0004	6.7e-8
2020-05-31	28.9133	28.9162	28.9162	3.2e-8
2020-06-01	22.5244	22.5676	22.5676	1.6e-7
2020-06-02	24.1392	24.0618	24.0618	1.1e-7
2020-06-03	35.0420	34.9857	34.9857	6.8e-8
2020-06-04	34.9155	35.0004	35.0004	6.7e-8
2020-06-05	48.6166	48.6611	48.6611	2.9e-7
2020-06-06	41.1237	41.0656	41.0656	1.6e-7
2020-06-07	30.0765	30.1174	30.1174	3.3e-8
2020-06-08	21.2425	21.3664	21.3664	1.5e-7
2020-06-09	35.0127	35.0004	35.0004	6.7e-8
2020-06-10	35.0493	34.9857	34.9857	6.8e-8
2020-06-11	45.8875	45.9458	45.9458	2.4e-7
2020-06-12	48.6189	48.6611	48.6611	2.9e-7
2020-06-13	42.3732	42.2668	42.2668	1.6e-7
2020-06-14	35.0547	35.0004	35.0004	6.7e-8
2020-06-15	21.3063	21.3664	21.3664	1.5e-7
2020-06-16	24.0887	24.0618	24.0618	1.1e-7
2020-06-17	35.0988	34.9857	34.9857	6.8e-8
2020-06-18	45.9240	45.9458	45.9458	2.4e-7
2020-06-19	36.1239	36.2016	36.2016	6.6e-8
2020-06-20	41.1050	41.0656	41.0656	1.6e-7
2020-06-21	28.9482	28.9162	28.9162	3.2e-8
2020-06-22	21.2484	21.3664	21.3664	1.5e-7
2020-06-23	23.9899	24.0618	24.0618	1.1e-7
2020-06-24	35.0514	35.0004	35.0004	6.7e-8
2020-06-25	47.1372	47.1470	47.1470	2.4e-7
2020-06-26	48.5686	48.6611	48.6611	2.9e-7
2020-06-27	41.1071	41.0656	41.0656	1.6e-7
2020-06-28	29.0247	28.9162	28.9162	3.2e-8
2020-06-29	34.9904	35.0004	35.0004	6.7e-8
2020-06-30	24.0292	24.0618	24.0618	1.1e-7
2020-07-01	36.2926	36.1869	36.1869	6.7e-8
2020-07-02	46.0073	45.9458	45.9458	2.4e-7
2020-07-03	48.5827	48.6611	48.6611	2.9e-7
2020-07-04	34.9774	35.0004	35.0004	6.7e-8
2020-07-05	28.9860	28.9162	28.9162	3.2e-8
2020-07-06	21.3124	21.3664	21.3664	1.5e-7
2020-07-07	25.1427	25.2631	25.2631	1.1e-7
2020-07-08	34.9999	34.9857	34.9857	6.8e-8
2020-07-09	35.0393	35.0004	35.0004	6.7e-8
2020-07-10	48.5786	48.6611	48.6611	2.9e-7
2020-07-11	41.0239	41.0656	41.0656	1.6e-7
2020-07-12	29.0086	28.9162	28.9162	3.2e-8
2020-07-13	22.6022	22.5676	22.5676	1.6e-7
2020-07-14	34.9539	35.0004	35.0004	6.7e-8
2020-07-15	35.0337	34.9857	34.9857	6.8e-8
2020-07-16	46.0561	45.9458	45.9458	2.4e-7
2020-07-17	48.6472	48.6611	48.6611	2.9e-7
2020-07-18	41.0122	41.0656	41.0656	1.6e-7
2020-07-19	36.2416	36.2016	36.2016	6.6e-8
2020-07-20	21.3889	21.3664	21.3664	1.5e-7
2020-07-21	23.9604	24.0618	24.0618	1.1e-7
2020-07-22	34.9296	34.9857	34.9857	6.8e-8
2020-07-23	45.9880	45.9458	45.9458	2.4e-7
2020-07-24	34.9840	35.0004	35.0004	6.7e-8
2020-07-25	42.1779	42.2668	42.2668	1.6e-7
2020-07-26	28.9370	28.9162	28.9162	3.2e-8
2020-07-27	21.4383	21.3664	21.3664	1.5e-7
2020-07-28	24.0404	24.0618	24.0618	1.1e-7
2020-07-29	34.9689	35.0004	35.0004	6.7e-8
2020-07-30	46.0389	45.9458	45.9458	2.4e-7
2020-07-31	49.9255	49.8624	49.8624	2.9e-7
2020-08-01	41.0247	41.0656	41.0656	1.6e-7
2020-08-02	28.9142	28.9162	28.9162	3.2e-8
2020-08-03	35.0753	35.0004	35.0004	6.7e-8
2020-08-04	24.0327	24.0618	24.0618	1.1e-7
2020-08-05	34.8932	34.9857	34.9857	6.8e-8
2020-08-06	47.1400	47.1470	47.1470	2.4e-7
2020-08-07	48.6840	48.6611	48.6611	2.9e-7
2020-08-08	34.9210	35.0004	35.0004	6.7e-8
2020-08-09	28.8553	28.9162	28.9162	3.2e-8
2020-08-10	21.4151	21.3664	21.3664	1.5e-7
2020-08-11	24.0965	24.0618	24.0618	1.1e-7
2020-08-12	36.1457	36.1869	36.1869	6.7e-8
2020-08-13	35.0253	35.0004	35.0004	6.7e-8
2020-08-14	48.7637	48.6611	48.6611	2.9e-7
2020-08-15	41.0876	41.0656	41.0656	1.6e-7
2020-08-16	28.8756	28.9162	28.9162	3.2e-8
2020-08-17	21.4046	21.3664	21.3664	1.5e-7
2020-08-18	36.2570	36.2016	36.2016	6.6e-8
2020-08-19	34.9214	34.9857	34.9857	6.8e-8
2020-08-20	45.8771	45.9458	45.9458	2.4e-7
2020-08-21	48.6894	48.6611	48.6611	2.9e-7
2020-08-22	41.0568	41.0656	41.0656	1.6e-7
2020-08-23	34.8913	35.0004	35.0004	6.7e-8
2020-08-24	22.5415	22.5676	22.5676	1.6e-7
2020-08-25	24.1269	24.0618	24.0618	1.1e-7
2020-08-26	34.9766	34.9857	34.9857	6.8e-8
2020-08-27	45.9018	45.9458	45.9458	2.4e-7
2020-08-28	35.0860	35.0004	35.0004	6.7e-8
2020-06-13	42.3732	42.2682	42.2682	1.7e-7
2020-06-14	35.0547	35.0005	35.0005	3.3e-8
2020-06-15	21.3063	21.3589	21.3589	8.4e-9
2020-06-16	24.0887	24.0659	24.0659	9.9e-8
2020-06-17	35.0988	34.9940	34.9940	1.1e-7
2020-06-18	45.9240	45.9430	45.9430	1.0e-8
2020-06-19	36.1239	36.1988	36.1988	1.7e-8
2020-06-20	41.1050	41.0698	41.0698	1.8e-7
2020-06-21	28.9482	28.9141	28.9141	1.3e-7
2020-06-22	21.2484	21.3589	21.3589	8.4e-9
2020-06-23	23.9899	24.0659	24.0659	9.9e-8
2020-06-24	35.0514	35.0005	35.0005	3.3e-8
2020-06-25	47.1372	47.1413	47.1413	2.6e-8
2020-06-26	48.5686	48.6578	48.6578	1.2e-7
2020-06-27	41.1071	41.0698	41.0698	1.8e-7
2020-06-28	29.0247	28.9141	28.9141	1.3e-7
2020-06-29	34.9904	35.0005	35.0005	3.3e-8
2020-06-30	24.0292	24.0659	24.0659	9.9e-8
2020-07-01	36.2926	36.1924	36.1924	1.2e-7
2020-07-02	46.0073	45.9430	45.9430	1.0e-8
2020-07-03	48.5827	48.6578	48.6578	1.2e-7
2020-07-04	34.9774	35.0005	35.0005	3.3e-8
2020-07-05	28.9860	28.9141	28.9141	1.3e-7
2020-07-06	21.3124	21.3589	21.3589	8.4e-9
2020-07-07	25.1427	25.2643	25.2643	1.1e-7
2020-07-08	34.9999	34.9940	34.9940	1.1e-7
2020-07-09	35.0393	35.0005	35.0005	3.3e-8
2020-07-10	48.5786	48.6578	48.6578	1.2e-7
2020-07-11	41.0239	41.0698	41.0698	1.8e-7
2020-07-12	29.0086	28.9141	28.9141	1.3e-7
2020-07-13	22.6022	22.5573	22.5573	7.3e-9
2020-07-14	34.9539	35.0005	35.0005	3.3e-8
2020-07-15	35.0337	34.9940	34.9940	1.1e-7
2020-07-16	46.0561	45.9430	45.9430	1.0e-8
2020-07-17	48.6472	48.6578	48.6578	1.2e-7
2020-07-18	41.0122	41.0698	41.0698	1.8e-7
2020-07-19	36.2416	36.1988	36.1988	1.7e-8
2020-07-20	21.3889	21.3589	21.3589	8.4e-9
2020-07-21	23.9604	24.0659	24.0659	9.9e-8
2020-07-22	34.9296	34.9940	34.9940	1.1e-7
2020-07-23	45.9880	45.9430	45.9430	1.0e-8
2020-07-24	34.9840	35.0005	35.0005	3.3e-8
2020-07-25	42.1779	42.2682	42.2682	1.7e-7
2020-07-26	28.9370	28.9141	28.9141	1.3e-7
2020-07-27	21.4383	21.3589	21.3589	8.4e-9
2020-07-28	24.0404	24.0659	24.0659	9.9e-8
2020-07-29	34.9689	35.0005	35.0005	3.3e-8
2020-07-30	46.0389	45.9430	45.9430	1.0e-8
2020-07-31	49.9255	49.8561	49.8561	1.0e-7
2020-08-01	41.0247	41.0698	41.0698	1.8e-7
2020-08-02	28.9142	28.9141	28.9141	1.3e-7
2020-08-03	35.0753	35.0005	35.0005	3.3e-8
2020-08-04	24.0327	24.0659	24.0659	9.9e-8
2020-08-05	34.8932	34.9940	34.9940	1.1e-7
2020-08-06	47.1400	47.1413	47.1413	2.6e-8
2020-08-07	48.6840	48.6578	48.6578	1.2e-7
2020-08-08	34.9210	35.0005	35.0005	3.3e-8
2020-08-09	28.8553	28.9141	28.9141	1.3e-7
2020-08-10	21.4151	21.3589	21.3589	8.4e-9
2020-08-11	24.0965	24.0659	24.0659	9.9e-8
2020-08-12	36.1457	36.1924	36.1924	1.2e-7
2020-08-13	35.0253	35.0005	35.0005	3.3e-8
2020-08-14	48.7637	48.6578	48.6578	1.2e-7
2020-08-15	41.0876	41.0698	41.0698	1.8e-7
2020-08-16	28.8756	28.9141	28.9141	1.3e-7
2020-08-17	21.4046	21.3589	21.3589	8.4e-9
2020-08-18	36.2570	36.1988	36.1988	1.7e-8
2020-08-19	34.9214	34.9940	34.9940	1.1e-7
2020-08-20	45.8771	45.9430	45.9430	1.0e-8
2020-08-21	48.6894	48.6578	48.6578	1.2e-7
2020-08-22	41.0568	41.0698	41.0698	1.8e-7
2020-08-23	34.8913	35.0005	35.0005	3.3e-8
2020-08-24	22.5415	22.5573	22.5573	7.3e-9
2020-08-25	24.1269	24.0659	24.0659	9.9e-8
2020-08-26	34.9766	34.9940	34.9940	1.1e-7
2020-08-27	45.9018	45.9430	45.9430	1.0e-8
2020-08-28	35.0860	35.0005	35.0005	3.3e-8
2020-08-29	41.1585	41.0698	41.0698	1.8e-7
2020-08-30	30.0879	30.1124	30.1124	1.2e-7
2020-08-31	21.3487	21.3589	21.3589	8.4e-9
2020-09-01	24.1452	24.0659	24.0659	9.9e-8
2020-09-02	34.9994	35.0005	35.0005	3.3e-8
2020-09-03	45.8503	45.9430	45.9430	1.0e-8
2020-09-04	48.6453	48.6578	48.6578	1.2e-7
2020-09-05	42.3122	42.2682	42.2682	1.7e-7
2020-09-06	28.8445	28.9141	28.9141	1.3e-7
2020-09-07	34.9139	35.0005	35.0005	3.3e-8
2020-09-08	24.0999	24.0659	24.0659	9.9e-8
2020-09-09	35.0306	34.9940	34.9940	1.1e-7
2020-09-10	45.8784	45.9430	45.9430	1.0e-8
2020-09-11	49.8590	49.8561	49.8561	1.0e-7

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Compute metrics	462 µs	5.61 ms	Not compared: different work
Compute metrics by calendar month	376 µs	5.05 ms	Not compared: different work
Compute custom metrics	1.19 ms	10.1 ms	Not compared: different work
Compute MASE	2.00 ms	5.62 ms	Not compared: different work

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

• Compute metrics, Compute metrics by calendar month, Compute custom metrics, Compute MASE: Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 7862400000,
    "cutoffs": [
      "2020-05-01T00:00:00.000Z",
      "2020-05-29T00:00:00.000Z",
      "2020-06-12T00:00:00.000Z"
    ]
  },
  "metricAggregation": "rolling"
}