---
url: >-
  https://jer-k.github.io/effect-prophet/benchmarks/flat/evaluation-flat-large-months-metrics.md
---

# Monthly, custom and scaled metrics

 [← All flat growth benchmarks](/benchmarks/flat)

Three 91-day folds scored by calendar month, with custom metrics and with MASE, in rolling windows. The scores Effect gives Python's own forecasts must equal Python's exactly. Python Prophet has no MASE, so Python registers its own; each fold's scale must match Effect's.

## Input

[`benchmark/inputs/v1/flat-mixed-components-large.json`](https://github.com/jer-k/effect-prophet/blob/main/benchmark/inputs/v1/flat-mixed-components-large.json): 256 history rows and 64 forecast rows.

## Output

Largest difference: **4.1e-7** (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

| Date | Actual | Effect Prophet | Python Prophet | Difference |
| --- | ---: | ---: | ---: | ---: |
| 2020-05-02 | 42.2518 | 42.2628 | 42.2628 | 1.5e-7 |
| 2020-05-03 | 28.8913 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-05-04 | 21.4235 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-05-05 | 35.0208 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-05-06 | 34.8995 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-05-07 | 45.9081 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-05-08 | 49.8937 | 49.8545 | 49.8545 | 2.8e-7 |
| 2020-05-09 | 41.0225 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-05-10 | 34.9022 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-05-11 | 21.3819 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-05-12 | 24.1165 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-05-13 | 34.9543 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-05-14 | 47.1330 | 47.1587 | 47.1587 | 4.0e-7 |
| 2020-05-15 | 35.1086 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-05-16 | 41.1251 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-05-17 | 28.8750 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-05-18 | 21.3793 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-05-19 | 24.1368 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-05-20 | 36.1571 | 36.2001 | 36.2001 | 2.6e-7 |
| 2020-05-21 | 45.8576 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-05-22 | 48.6729 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-05-23 | 41.0880 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-05-24 | 28.8224 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-05-25 | 34.9489 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-05-26 | 25.3202 | 25.2464 | 25.2464 | 3.1e-7 |
| 2020-05-27 | 35.0023 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-05-28 | 45.8819 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-05-29 | 48.7001 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-05-30 | 35.1050 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-05-31 | 28.9133 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-06-01 | 22.5244 | 22.5596 | 22.5596 | 1.7e-7 |
| 2020-06-02 | 24.1392 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-06-03 | 35.0420 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-06-04 | 34.9155 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-06-05 | 48.6166 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-06-06 | 41.1237 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-06-07 | 30.0765 | 30.1258 | 30.1258 | 9.7e-8 |
| 2020-06-08 | 21.2425 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-06-09 | 35.0127 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-06-10 | 35.0493 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-06-11 | 45.8875 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-06-12 | 48.6189 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-06-13 | 42.3732 | 42.2628 | 42.2628 | 1.5e-7 |
| 2020-06-14 | 35.0547 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-06-15 | 21.3063 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-06-16 | 24.0887 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-06-17 | 35.0988 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-06-18 | 45.9240 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-06-19 | 36.1239 | 36.2001 | 36.2001 | 2.6e-7 |
| 2020-06-20 | 41.1050 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-06-21 | 28.9482 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-06-22 | 21.2484 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-06-23 | 23.9899 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-06-24 | 35.0514 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-06-25 | 47.1372 | 47.1587 | 47.1587 | 4.0e-7 |
| 2020-06-26 | 48.5686 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-06-27 | 41.1071 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-06-28 | 29.0247 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-06-29 | 34.9904 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-06-30 | 24.0292 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-07-01 | 36.2926 | 36.1932 | 36.1932 | 4.1e-7 |
| 2020-07-02 | 46.0073 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-07-03 | 48.5827 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-07-04 | 34.9774 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-07-05 | 28.9860 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-07-06 | 21.3124 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-07-07 | 25.1427 | 25.2464 | 25.2464 | 3.1e-7 |
| 2020-07-08 | 34.9999 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-07-09 | 35.0393 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-07-10 | 48.5786 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-07-11 | 41.0239 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-07-12 | 29.0086 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-07-13 | 22.6022 | 22.5596 | 22.5596 | 1.7e-7 |
| 2020-07-14 | 34.9539 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-07-15 | 35.0337 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-07-16 | 46.0561 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-07-17 | 48.6472 | 48.6566 | 48.6566 | 2.6e-7 |
| 2020-07-18 | 41.0122 | 41.0648 | 41.0648 | 1.3e-7 |
| 2020-07-19 | 36.2416 | 36.2001 | 36.2001 | 2.6e-7 |
| 2020-07-20 | 21.3889 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-07-21 | 23.9604 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-07-22 | 34.9296 | 34.9953 | 34.9953 | 3.9e-7 |
| 2020-07-23 | 45.9880 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-07-24 | 34.9840 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-07-25 | 42.1779 | 42.2628 | 42.2628 | 1.5e-7 |
| 2020-07-26 | 28.9370 | 28.9279 | 28.9279 | 7.7e-8 |
| 2020-07-27 | 21.4383 | 21.3616 | 21.3616 | 1.5e-7 |
| 2020-07-28 | 24.0404 | 24.0484 | 24.0484 | 2.9e-7 |
| 2020-07-29 | 34.9689 | 35.0022 | 35.0022 | 2.4e-7 |
| 2020-07-30 | 46.0389 | 45.9608 | 45.9608 | 3.8e-7 |
| 2020-07-31 | 49.9255 | 49.8545 | 49.8545 | 2.8e-7 |
| 2020-05-30 | 35.1050 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-05-31 | 28.9133 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-06-01 | 22.5244 | 22.5676 | 22.5676 | 1.6e-7 |
| 2020-06-02 | 24.1392 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-06-03 | 35.0420 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-06-04 | 34.9155 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-06-05 | 48.6166 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-06-06 | 41.1237 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-06-07 | 30.0765 | 30.1174 | 30.1174 | 3.3e-8 |
| 2020-06-08 | 21.2425 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-06-09 | 35.0127 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-06-10 | 35.0493 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-06-11 | 45.8875 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-06-12 | 48.6189 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-06-13 | 42.3732 | 42.2668 | 42.2668 | 1.6e-7 |
| 2020-06-14 | 35.0547 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-06-15 | 21.3063 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-06-16 | 24.0887 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-06-17 | 35.0988 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-06-18 | 45.9240 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-06-19 | 36.1239 | 36.2016 | 36.2016 | 6.6e-8 |
| 2020-06-20 | 41.1050 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-06-21 | 28.9482 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-06-22 | 21.2484 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-06-23 | 23.9899 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-06-24 | 35.0514 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-06-25 | 47.1372 | 47.1470 | 47.1470 | 2.4e-7 |
| 2020-06-26 | 48.5686 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-06-27 | 41.1071 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-06-28 | 29.0247 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-06-29 | 34.9904 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-06-30 | 24.0292 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-07-01 | 36.2926 | 36.1869 | 36.1869 | 6.7e-8 |
| 2020-07-02 | 46.0073 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-07-03 | 48.5827 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-07-04 | 34.9774 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-07-05 | 28.9860 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-07-06 | 21.3124 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-07-07 | 25.1427 | 25.2631 | 25.2631 | 1.1e-7 |
| 2020-07-08 | 34.9999 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-07-09 | 35.0393 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-07-10 | 48.5786 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-07-11 | 41.0239 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-07-12 | 29.0086 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-07-13 | 22.6022 | 22.5676 | 22.5676 | 1.6e-7 |
| 2020-07-14 | 34.9539 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-07-15 | 35.0337 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-07-16 | 46.0561 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-07-17 | 48.6472 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-07-18 | 41.0122 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-07-19 | 36.2416 | 36.2016 | 36.2016 | 6.6e-8 |
| 2020-07-20 | 21.3889 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-07-21 | 23.9604 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-07-22 | 34.9296 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-07-23 | 45.9880 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-07-24 | 34.9840 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-07-25 | 42.1779 | 42.2668 | 42.2668 | 1.6e-7 |
| 2020-07-26 | 28.9370 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-07-27 | 21.4383 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-07-28 | 24.0404 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-07-29 | 34.9689 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-07-30 | 46.0389 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-07-31 | 49.9255 | 49.8624 | 49.8624 | 2.9e-7 |
| 2020-08-01 | 41.0247 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-08-02 | 28.9142 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-08-03 | 35.0753 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-08-04 | 24.0327 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-08-05 | 34.8932 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-08-06 | 47.1400 | 47.1470 | 47.1470 | 2.4e-7 |
| 2020-08-07 | 48.6840 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-08-08 | 34.9210 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-08-09 | 28.8553 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-08-10 | 21.4151 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-08-11 | 24.0965 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-08-12 | 36.1457 | 36.1869 | 36.1869 | 6.7e-8 |
| 2020-08-13 | 35.0253 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-08-14 | 48.7637 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-08-15 | 41.0876 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-08-16 | 28.8756 | 28.9162 | 28.9162 | 3.2e-8 |
| 2020-08-17 | 21.4046 | 21.3664 | 21.3664 | 1.5e-7 |
| 2020-08-18 | 36.2570 | 36.2016 | 36.2016 | 6.6e-8 |
| 2020-08-19 | 34.9214 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-08-20 | 45.8771 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-08-21 | 48.6894 | 48.6611 | 48.6611 | 2.9e-7 |
| 2020-08-22 | 41.0568 | 41.0656 | 41.0656 | 1.6e-7 |
| 2020-08-23 | 34.8913 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-08-24 | 22.5415 | 22.5676 | 22.5676 | 1.6e-7 |
| 2020-08-25 | 24.1269 | 24.0618 | 24.0618 | 1.1e-7 |
| 2020-08-26 | 34.9766 | 34.9857 | 34.9857 | 6.8e-8 |
| 2020-08-27 | 45.9018 | 45.9458 | 45.9458 | 2.4e-7 |
| 2020-08-28 | 35.0860 | 35.0004 | 35.0004 | 6.7e-8 |
| 2020-06-13 | 42.3732 | 42.2682 | 42.2682 | 1.7e-7 |
| 2020-06-14 | 35.0547 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-06-15 | 21.3063 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-06-16 | 24.0887 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-06-17 | 35.0988 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-06-18 | 45.9240 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-06-19 | 36.1239 | 36.1988 | 36.1988 | 1.7e-8 |
| 2020-06-20 | 41.1050 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-06-21 | 28.9482 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-06-22 | 21.2484 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-06-23 | 23.9899 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-06-24 | 35.0514 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-06-25 | 47.1372 | 47.1413 | 47.1413 | 2.6e-8 |
| 2020-06-26 | 48.5686 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-06-27 | 41.1071 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-06-28 | 29.0247 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-06-29 | 34.9904 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-06-30 | 24.0292 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-07-01 | 36.2926 | 36.1924 | 36.1924 | 1.2e-7 |
| 2020-07-02 | 46.0073 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-07-03 | 48.5827 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-07-04 | 34.9774 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-07-05 | 28.9860 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-07-06 | 21.3124 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-07-07 | 25.1427 | 25.2643 | 25.2643 | 1.1e-7 |
| 2020-07-08 | 34.9999 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-07-09 | 35.0393 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-07-10 | 48.5786 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-07-11 | 41.0239 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-07-12 | 29.0086 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-07-13 | 22.6022 | 22.5573 | 22.5573 | 7.3e-9 |
| 2020-07-14 | 34.9539 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-07-15 | 35.0337 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-07-16 | 46.0561 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-07-17 | 48.6472 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-07-18 | 41.0122 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-07-19 | 36.2416 | 36.1988 | 36.1988 | 1.7e-8 |
| 2020-07-20 | 21.3889 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-07-21 | 23.9604 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-07-22 | 34.9296 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-07-23 | 45.9880 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-07-24 | 34.9840 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-07-25 | 42.1779 | 42.2682 | 42.2682 | 1.7e-7 |
| 2020-07-26 | 28.9370 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-07-27 | 21.4383 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-07-28 | 24.0404 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-07-29 | 34.9689 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-07-30 | 46.0389 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-07-31 | 49.9255 | 49.8561 | 49.8561 | 1.0e-7 |
| 2020-08-01 | 41.0247 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-08-02 | 28.9142 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-08-03 | 35.0753 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-08-04 | 24.0327 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-08-05 | 34.8932 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-08-06 | 47.1400 | 47.1413 | 47.1413 | 2.6e-8 |
| 2020-08-07 | 48.6840 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-08-08 | 34.9210 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-08-09 | 28.8553 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-08-10 | 21.4151 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-08-11 | 24.0965 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-08-12 | 36.1457 | 36.1924 | 36.1924 | 1.2e-7 |
| 2020-08-13 | 35.0253 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-08-14 | 48.7637 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-08-15 | 41.0876 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-08-16 | 28.8756 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-08-17 | 21.4046 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-08-18 | 36.2570 | 36.1988 | 36.1988 | 1.7e-8 |
| 2020-08-19 | 34.9214 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-08-20 | 45.8771 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-08-21 | 48.6894 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-08-22 | 41.0568 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-08-23 | 34.8913 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-08-24 | 22.5415 | 22.5573 | 22.5573 | 7.3e-9 |
| 2020-08-25 | 24.1269 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-08-26 | 34.9766 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-08-27 | 45.9018 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-08-28 | 35.0860 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-08-29 | 41.1585 | 41.0698 | 41.0698 | 1.8e-7 |
| 2020-08-30 | 30.0879 | 30.1124 | 30.1124 | 1.2e-7 |
| 2020-08-31 | 21.3487 | 21.3589 | 21.3589 | 8.4e-9 |
| 2020-09-01 | 24.1452 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-09-02 | 34.9994 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-09-03 | 45.8503 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-09-04 | 48.6453 | 48.6578 | 48.6578 | 1.2e-7 |
| 2020-09-05 | 42.3122 | 42.2682 | 42.2682 | 1.7e-7 |
| 2020-09-06 | 28.8445 | 28.9141 | 28.9141 | 1.3e-7 |
| 2020-09-07 | 34.9139 | 35.0005 | 35.0005 | 3.3e-8 |
| 2020-09-08 | 24.0999 | 24.0659 | 24.0659 | 9.9e-8 |
| 2020-09-09 | 35.0306 | 34.9940 | 34.9940 | 1.1e-7 |
| 2020-09-10 | 45.8784 | 45.9430 | 45.9430 | 1.0e-8 |
| 2020-09-11 | 49.8590 | 49.8561 | 49.8561 | 1.0e-7 |

## Timing

Median of all measured repetitions.

| Step | Effect Prophet | Python Prophet | Effect Prophet is |
| --- | ---: | ---: | --- |
| Compute metrics | 462 µs | 5.61 ms | Not compared: different work |
| Compute metrics by calendar month | 376 µs | 5.05 ms | Not compared: different work |
| Compute custom metrics | 1.19 ms | 10.1 ms | Not compared: different work |
| Compute MASE | 2.00 ms | 5.62 ms | Not compared: different work |

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

* **Compute metrics, Compute metrics by calendar month, Compute custom metrics, Compute MASE:** Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

## Settings

The case declaration both libraries ran.

```json
{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 7862400000,
    "cutoffs": [
      "2020-05-01T00:00:00.000Z",
      "2020-05-29T00:00:00.000Z",
      "2020-06-12T00:00:00.000Z"
    ]
  },
  "metricAggregation": "rolling"
}
```