---
url: >-
  https://jer-k.github.io/effect-prophet/benchmarks/flat/evaluation-flat-mixed-search.md
---

# Choosing settings and a final holdout

 [← All flat growth benchmarks](/benchmarks/flat)

Three candidate settings compared on the same folds, then a one-time holdout check.

## Input

[`benchmark/inputs/v1/flat-mixed-components.json`](https://github.com/jer-k/effect-prophet/blob/main/benchmark/inputs/v1/flat-mixed-components.json): 96 history rows and 24 forecast rows.

## Output

Largest difference: **2.8e-7** (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

| Date | Actual | Effect Prophet | Python Prophet | Difference |
| --- | ---: | ---: | ---: | ---: |
| 2020-03-04 | 35.0775 | 34.9780 | 34.9780 | 1.2e-7 |
| 2020-03-05 | 45.8872 | 45.9405 | 45.9405 | 3.0e-8 |
| 2020-03-06 | 34.9423 | 35.0019 | 35.0019 | 1.0e-7 |
| 2020-03-07 | 41.1218 | 41.0645 | 41.0645 | 7.7e-9 |
| 2020-03-08 | 28.9133 | 28.8939 | 28.8939 | 1.7e-7 |
| 2020-03-09 | 22.4365 | 22.5775 | 22.5775 | 2.8e-7 |
| 2020-03-10 | 24.0283 | 24.0826 | 24.0826 | 2.6e-7 |
| 2020-03-11 | 35.0568 | 35.0021 | 35.0021 | 1.0e-7 |
| 2020-03-12 | 45.9078 | 45.9398 | 45.9398 | 9.8e-8 |
| 2020-03-13 | 48.5866 | 48.6775 | 48.6775 | 2.6e-8 |
| 2020-03-14 | 41.1455 | 41.0759 | 41.0759 | 1.0e-8 |
| 2020-03-15 | 30.2075 | 30.0881 | 30.0881 | 4.7e-8 |
| 2020-03-16 | 34.9665 | 35.0021 | 35.0021 | 1.0e-7 |
| 2020-03-17 | 24.0576 | 24.0795 | 24.0795 | 1.9e-7 |
| 2020-03-18 | 35.1048 | 34.9897 | 34.9897 | 3.0e-9 |
| 2020-03-19 | 45.9668 | 45.9330 | 45.9330 | 8.4e-8 |
| 2020-03-20 | 48.5708 | 48.6675 | 48.6675 | 5.7e-8 |
| 2020-03-21 | 36.2070 | 36.2037 | 36.2037 | 5.4e-8 |
| 2020-03-22 | 28.9741 | 28.9071 | 28.9071 | 1.7e-7 |
| 2020-03-23 | 21.2751 | 21.3559 | 21.3559 | 2.0e-7 |
| 2020-03-24 | 23.9598 | 24.0746 | 24.0746 | 1.2e-7 |

## Timing

Median of all measured repetitions.

| Step | Effect Prophet | Python Prophet | Effect Prophet is |
| --- | ---: | ---: | --- |
| Search settings | 12.3 ms | 146 ms | Not compared: different work |
| Final holdout check | 2.02 ms | 13.6 ms | Not compared: different work |
| Save report | 277 µs | 9 µs | Not compared: different work |
| Load report | 308 µs | 5 µs | Not compared: different work |

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

* **Search settings, Final holdout check, Save report, Load report:** Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

## Settings

The case declaration both libraries ran.

```json
{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "flat",
    "scaling": "minmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10,
        "conditionName": "active",
        "mode": "multiplicative"
      }
    ],
    "events": [
      {
        "name": "campaign",
        "date": "2020-01-22",
        "lowerWindowDays": 0,
        "upperWindowDays": 0,
        "priorScale": 10
      }
    ],
    "regressors": [
      {
        "name": "promotion",
        "priorScale": 10,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "lbfgs",
    "maxIterations": 10000,
    "fallback": "none",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "LBFGS",
    "maxIterations": 10000,
    "newtonFallback": false
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-03T00:00:00.000Z",
      "2020-03-10T00:00:00.000Z",
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "candidates": [
    {
      "id": "prior-low",
      "changepointPriorScale": 0.05
    },
    {
      "id": "prior-high",
      "changepointPriorScale": 0.2
    },
    {
      "id": "prior-third",
      "changepointPriorScale": 0.5
    }
  ],
  "holdoutRows": 8
}
```