Choosing settings and a final holdout

match ← All logistic growth benchmarks

Two candidate settings compared on the same fold, then a one-time holdout check.

Input

benchmark/inputs/v1/logistic-implicit-floor.json: 96 history rows and 24 forecast rows.

Output

Largest difference: 1.6e-11 (< 0.0001% of the input's range).

Each row is one forecast made during cross-validation, next to the value that actually happened.

Date	Actual	Effect Prophet	Python Prophet	Difference
2020-03-18	56.3482	56.2290	56.2290	1.6e-11
2020-03-19	57.1831	57.1500	57.1500	1.6e-11
2020-03-20	57.7257	57.8237	57.8237	1.5e-11
2020-03-21	58.1108	58.1079	58.1079	1.5e-11
2020-03-22	58.2935	58.2238	58.2238	1.4e-11
2020-03-23	58.4460	58.5260	58.5260	1.4e-11
2020-03-24	59.0232	59.1345	59.1345	1.4e-11

Timing

Median of all measured repetitions.

Step	Effect Prophet	Python Prophet	Effect Prophet is
Search settings	36.8 ms	123 ms	3.3× faster
Final holdout check	17.8 ms	31.0 ms	1.7× faster
Save report	256 µs	9 µs	29× slower
Load report	171 µs	4 µs	45× slower

Settings

The case declaration both libraries ran.

{
  "kind": "evaluation",
  "comparison": {
    "kind": "different-public-work",
    "evidenceId": "rolling-origin-evaluation-v1"
  },
  "configuration": {
    "growth": "logistic",
    "scaling": "absmax",
    "seasonalityMode": "additive",
    "holidaysMode": "additive",
    "changepoints": {
      "mode": "explicit",
      "timestamps": [
        "2020-02-10T00:00:00.000Z"
      ]
    },
    "changepointPriorScale": 0.2,
    "seasonalities": [
      {
        "name": "weekly-custom",
        "periodDays": 7,
        "fourierOrder": 2,
        "priorScale": 10
      }
    ],
    "events": [],
    "regressors": []
  },
  "pythonOptimizer": {
    "algorithm": "Auto",
    "maxIterations": 10000,
    "newtonFallback": true,
    "sigFigs": 12
  },
  "plan": {
    "horizonMs": 604800000,
    "cutoffs": [
      "2020-03-17T00:00:00.000Z"
    ]
  },
  "metricAggregation": "horizons",
  "mapeZeroActual": "exclude",
  "candidates": [
    {
      "id": "prior-low",
      "changepointPriorScale": 0.2
    },
    {
      "id": "prior-high",
      "changepointPriorScale": 0.25
    }
  ],
  "holdoutRows": 8
}