---
url: >-
  https://jer-k.github.io/effect-prophet/benchmarks/linear/evaluation-linear-regressor-model-point.md
---

# Cross-validation with a forecast regressor

 [← All linear growth benchmarks](/benchmarks/linear)

Two cutoffs and a 7-day horizon, fitting the regressor model again in each fold. Python's cross_validation refits it on standardized values, so Python is fitted at each cutoff instead.

## Input

[`benchmark/inputs/v1/map-regressors-medium.json`](https://github.com/jer-k/effect-prophet/blob/main/benchmark/inputs/v1/map-regressors-medium.json): 256 history rows and 64 forecast rows.

## Output

Largest difference: **0.0024** (0.0025% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

| Date | Actual | Effect Prophet | Python Prophet | Difference |
| --- | ---: | ---: | ---: | ---: |
| 2020-08-23 | 97.5341 | 97.6352 | 97.6368 | 0.0016 |
| 2020-08-24 | 98.5406 | 98.5658 | 98.5679 | 0.0020 |
| 2020-08-25 | 97.3639 | 97.3234 | 97.3254 | 0.0021 |
| 2020-08-26 | 97.8038 | 97.8828 | 97.8849 | 0.0020 |
| 2020-08-27 | 98.3144 | 98.4353 | 98.4373 | 0.0020 |
| 2020-08-28 | 99.3311 | 99.3387 | 99.3411 | 0.0024 |
| 2020-08-29 | 99.8524 | 99.8764 | 99.8788 | 0.0024 |
| 2020-09-03 | 102.6061 | 102.8260 | 102.8260 | 7.3e-5 |
| 2020-09-04 | 101.7963 | 101.9156 | 101.9153 | 0.00022 |
| 2020-09-05 | 102.7016 | 102.7640 | 102.7638 | 0.00022 |
| 2020-09-06 | 103.0832 | 103.2447 | 103.2445 | 0.00017 |
| 2020-09-07 | 103.5750 | 103.7194 | 103.7193 | 0.00012 |
| 2020-09-08 | 104.1993 | 104.1885 | 104.1884 | 7.6e-5 |
| 2020-09-09 | 106.8323 | 106.8125 | 106.8124 | 0.00010 |

## Timing

Median of all measured repetitions.

| Step | Effect Prophet | Python Prophet | Effect Prophet is |
| --- | ---: | ---: | --- |
| Convert input rows | 108 µs | 544 µs | Not compared: different work |
| Plan cross-validation folds | 2.25 ms | 796 µs | Not compared: different work |
| Cross-validate | 51.6 ms | 185 ms | Not compared: different work |
| Compute metrics | 58 µs | 5.50 ms | Not compared: different work |
| Score baselines | 2.90 ms | 1.42 ms | Not compared: different work |
| First cross-validation in a new process | 245 ms | 1.62 s | Not compared: different work |

Some steps aren't compared, because the two libraries' public calls produce different outputs. Their times show what each call costs, not which library is faster.

* **Convert input rows, Plan cross-validation folds, Cross-validate, Compute metrics, Score baselines, First cross-validation in a new process:** Python Prophet's cross-validation returns different outputs, and baselines, search and holdout reports have no Python API, so the benchmark times equivalent application code. The two libraries also compute ranges and some metrics differently.

## Settings

The case declaration both libraries ran.

```json
{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "linear",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "auto",
      "count": 25,
      "range": 0.8
    },
    "changepointPriorScale": 0.05,
    "seasonalities": [],
    "events": [],
    "regressors": [
      {
        "name": "binary-auto",
        "priorScale": 10,
        "standardization": "auto"
      },
      {
        "name": "numeric-auto",
        "priorScale": 10,
        "standardization": "auto",
        "predictor": true
      },
      {
        "name": "numeric-always",
        "priorScale": 7,
        "standardization": "always"
      },
      {
        "name": "numeric-never",
        "priorScale": 5,
        "standardization": "never"
      }
    ]
  },
  "effectOptimizer": {
    "algorithm": "auto",
    "maxIterations": 10000,
    "fallback": "newton",
    "lbfgs": {
      "historySize": 5,
      "initAlpha": 0.001,
      "tolObj": 1e-12,
      "tolRelObj": 10000,
      "tolGrad": 1e-8,
      "tolRelGrad": 10000000,
      "tolParam": 1e-8
    }
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-08-22T00:00:00.000Z",
      "2020-09-02T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons"
}
```