---
url: >-
  https://jer-k.github.io/effect-prophet/benchmarks/flat/evaluation-flat-large-parallel.md
---

# Twelve folds, sequential and parallel

 [← All flat growth benchmarks](/benchmarks/flat)

Twelve weekly cutoffs on the 256-day history. Python Prophet is also timed with `parallel="threads"` and `parallel="processes"`, which must give the same forecasts as running sequentially.

## Input

[`benchmark/inputs/v1/flat-mixed-components-large.json`](https://github.com/jer-k/effect-prophet/blob/main/benchmark/inputs/v1/flat-mixed-components-large.json): 256 history rows and 64 forecast rows.

## Output

Largest difference: **4.9e-7** (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

| Date | Actual | Effect Prophet | Python Prophet | Difference |
| --- | ---: | ---: | ---: | ---: |
| 2020-06-21 | 28.9482 | 28.9148 | 28.9148 | 2.5e-7 |
| 2020-06-22 | 21.2484 | 21.3545 | 21.3545 | 5.1e-8 |
| 2020-06-23 | 23.9899 | 24.0677 | 24.0677 | 2.8e-9 |
| 2020-06-24 | 35.0514 | 35.0015 | 35.0015 | 2.5e-7 |
| 2020-06-25 | 47.1372 | 47.1401 | 47.1401 | 3.5e-7 |
| 2020-06-26 | 48.5686 | 48.6578 | 48.6578 | 4.9e-7 |
| 2020-06-27 | 41.1071 | 41.0755 | 41.0755 | 4.4e-7 |
| 2020-06-28 | 29.0247 | 28.9169 | 28.9169 | 1.1e-7 |
| 2020-06-29 | 34.9904 | 35.0005 | 35.0005 | 1.1e-7 |
| 2020-06-30 | 24.0292 | 24.0635 | 24.0635 | 2.2e-7 |
| 2020-07-01 | 36.2926 | 36.1993 | 36.1993 | 1.6e-7 |
| 2020-07-02 | 46.0073 | 45.9411 | 45.9411 | 6.1e-8 |
| 2020-07-03 | 48.5827 | 48.6546 | 48.6546 | 6.7e-10 |
| 2020-07-04 | 34.9774 | 35.0005 | 35.0005 | 1.1e-7 |
| 2020-07-05 | 28.9860 | 28.9211 | 28.9211 | 2.1e-8 |
| 2020-07-06 | 21.3124 | 21.3492 | 21.3492 | 4.6e-8 |
| 2020-07-07 | 25.1427 | 25.2630 | 25.2630 | 5.5e-8 |
| 2020-07-08 | 34.9999 | 35.0036 | 35.0036 | 1.6e-7 |
| 2020-07-09 | 35.0393 | 35.0007 | 35.0007 | 9.4e-8 |
| 2020-07-10 | 48.5786 | 48.6504 | 48.6504 | 2.0e-7 |
| 2020-07-11 | 41.0239 | 41.0764 | 41.0764 | 9.0e-8 |
| 2020-07-12 | 29.0086 | 28.9247 | 28.9247 | 5.8e-8 |
| 2020-07-13 | 22.6022 | 22.5471 | 22.5471 | 4.6e-8 |
| 2020-07-14 | 34.9539 | 35.0004 | 35.0004 | 1.1e-7 |
| 2020-07-15 | 35.0337 | 35.0041 | 35.0041 | 1.5e-7 |
| 2020-07-16 | 46.0561 | 45.9451 | 45.9451 | 1.8e-7 |
| 2020-07-17 | 48.6472 | 48.6477 | 48.6477 | 1.6e-7 |
| 2020-07-18 | 41.0122 | 41.0756 | 41.0756 | 1.1e-7 |
| 2020-07-19 | 36.2416 | 36.2004 | 36.2004 | 1.8e-7 |
| 2020-07-20 | 21.3889 | 21.3513 | 21.3513 | 9.1e-8 |
| 2020-07-21 | 23.9604 | 24.0555 | 24.0555 | 3.3e-8 |
| 2020-07-22 | 34.9296 | 35.0053 | 35.0053 | 2.0e-7 |
| 2020-07-23 | 45.9880 | 45.9491 | 45.9491 | 4.3e-7 |
| 2020-07-24 | 34.9840 | 35.0011 | 35.0011 | 2.0e-7 |
| 2020-07-25 | 42.1779 | 42.2721 | 42.2721 | 3.0e-7 |
| 2020-07-26 | 28.9370 | 28.9275 | 28.9275 | 7.3e-8 |
| 2020-07-27 | 21.4383 | 21.3533 | 21.3533 | 1.4e-7 |
| 2020-07-28 | 24.0404 | 24.0521 | 24.0521 | 2.8e-7 |
| 2020-07-29 | 34.9689 | 35.0005 | 35.0005 | 2.3e-7 |
| 2020-07-30 | 46.0389 | 45.9514 | 45.9514 | 3.8e-7 |
| 2020-07-31 | 49.9255 | 49.8456 | 49.8456 | 2.5e-7 |
| 2020-08-01 | 41.0247 | 41.0694 | 41.0694 | 1.3e-7 |
| 2020-08-02 | 28.9142 | 28.9273 | 28.9273 | 5.0e-8 |
| 2020-08-03 | 35.0753 | 35.0010 | 35.0010 | 1.8e-7 |
| 2020-08-04 | 24.0327 | 24.0514 | 24.0514 | 2.5e-7 |
| 2020-08-05 | 34.8932 | 35.0010 | 35.0010 | 3.1e-7 |
| 2020-08-06 | 47.1400 | 47.1548 | 47.1548 | 2.7e-7 |
| 2020-08-07 | 48.6840 | 48.6493 | 48.6493 | 1.6e-7 |
| 2020-08-08 | 34.9210 | 35.0010 | 35.0010 | 1.8e-7 |
| 2020-08-09 | 28.8553 | 28.9266 | 28.9266 | 5.5e-9 |
| 2020-08-10 | 21.4151 | 21.3565 | 21.3565 | 1.1e-7 |
| 2020-08-11 | 24.0965 | 24.0501 | 24.0501 | 1.2e-7 |
| 2020-08-12 | 36.1457 | 36.1976 | 36.1976 | 7.1e-9 |
| 2020-08-13 | 35.0253 | 35.0004 | 35.0004 | 3.1e-8 |
| 2020-08-14 | 48.7637 | 48.6511 | 48.6511 | 1.9e-7 |
| 2020-08-15 | 41.0876 | 41.0671 | 41.0671 | 1.4e-7 |
| 2020-08-16 | 28.8756 | 28.9246 | 28.9246 | 3.3e-8 |
| 2020-08-17 | 21.4046 | 21.3588 | 21.3588 | 9.7e-8 |
| 2020-08-18 | 36.2570 | 36.1993 | 36.1993 | 2.2e-8 |
| 2020-08-19 | 34.9214 | 34.9954 | 34.9954 | 3.7e-8 |
| 2020-08-20 | 45.8771 | 45.9542 | 45.9542 | 1.1e-7 |
| 2020-08-21 | 48.6894 | 48.6558 | 48.6558 | 2.0e-7 |
| 2020-08-22 | 41.0568 | 41.0680 | 41.0680 | 1.6e-7 |
| 2020-08-23 | 34.8913 | 35.0006 | 35.0006 | 1.4e-7 |
| 2020-08-24 | 22.5415 | 22.5602 | 22.5602 | 1.1e-7 |
| 2020-08-25 | 24.1269 | 24.0532 | 24.0532 | 2.6e-8 |
| 2020-08-26 | 34.9766 | 34.9920 | 34.9920 | 2.1e-8 |
| 2020-08-27 | 45.9018 | 45.9518 | 45.9518 | 1.1e-7 |
| 2020-08-28 | 35.0860 | 35.0006 | 35.0006 | 1.4e-7 |
| 2020-08-29 | 41.1585 | 41.0676 | 41.0676 | 2.8e-7 |
| 2020-08-30 | 30.0879 | 30.1225 | 30.1225 | 2.0e-7 |
| 2020-08-31 | 21.3487 | 21.3596 | 21.3596 | 2.9e-7 |
| 2020-09-01 | 24.1452 | 24.0553 | 24.0553 | 2.6e-7 |
| 2020-09-02 | 34.9994 | 35.0010 | 35.0010 | 1.4e-7 |
| 2020-09-03 | 45.8503 | 45.9492 | 45.9492 | 2.4e-8 |
| 2020-09-04 | 48.6453 | 48.6573 | 48.6573 | 1.2e-8 |
| 2020-09-05 | 42.3122 | 42.2698 | 42.2698 | 6.3e-8 |
| 2020-09-06 | 28.8445 | 28.9217 | 28.9217 | 2.2e-7 |
| 2020-09-07 | 34.9139 | 35.0008 | 35.0008 | 5.1e-8 |
| 2020-09-08 | 24.0999 | 24.0584 | 24.0584 | 2.8e-8 |
| 2020-09-09 | 35.0306 | 34.9921 | 34.9921 | 1.2e-7 |
| 2020-09-10 | 45.8784 | 45.9457 | 45.9457 | 8.4e-8 |
| 2020-09-11 | 49.8590 | 49.8567 | 49.8567 | 4.7e-8 |
| 2020-09-12 | 35.1111 | 35.0008 | 35.0008 | 5.1e-8 |

## Timing

Median of all measured repetitions.

Effect Prophet runs cross-validation folds one at a time, so only Python Prophet has parallel timings.

| Step | Effect Prophet | Python Prophet | Effect Prophet is |
| --- | ---: | ---: | --- |
| Cross-validate | 27.6 ms | 184 ms | Not compared: different work |
| Cross-validate, Python `parallel="threads"` | — | 199 ms | — |
| Cross-validate, Python `parallel="processes"` | — | 1.50 s | — |

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

* **Cross-validate:** Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

## Settings

The case declaration both libraries ran.

```json
{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-06-20T00:00:00.000Z",
      "2020-06-27T00:00:00.000Z",
      "2020-07-04T00:00:00.000Z",
      "2020-07-11T00:00:00.000Z",
      "2020-07-18T00:00:00.000Z",
      "2020-07-25T00:00:00.000Z",
      "2020-08-01T00:00:00.000Z",
      "2020-08-08T00:00:00.000Z",
      "2020-08-15T00:00:00.000Z",
      "2020-08-22T00:00:00.000Z",
      "2020-08-29T00:00:00.000Z",
      "2020-09-05T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}
```