Cross-validation and choosing settings benchmarks

25 of 25 cases match Python Prophet.

Rolling-origin cross-validation, metrics, baselines, settings search and a final holdout check, on every trend type.

Each case runs the same input through Effect Prophet and Python Prophet 1.4.0, then compares every forecast row, the trend, each component and the fitted noise. A case matches when all of them agree within its tolerances, and only matching cases are timed.

Each case's dot shows its result: matches, matches with internal differences flagged, intentional difference, doesn't match. Hover over a dot for details.

How to read the tables:

• Effect output and Python output are each library's final forecast value. Open a case to see every row side by side.
• Largest difference is the biggest gap between the two forecasts, as a share of the input data's range.
• Effect time and Python time are the median time for the timed step, usually fitting and predicting. Open a case for every step, including first-run and save/load times.
• A timed step marked different work calls public operations that produce different outputs in the two libraries, so its times aren't a speed comparison. The case page explains the difference.

About these runs

Each trend's cases come from that trend's own run.

Linear growth

• Run 2026-11-07T174133-142Z-3a5a6cec on 2026-11-07, commit 3a5a6ce
• Both libraries ran in linux/arm64 containers on Apple M4 Pro
• effect-prophet 0.0.0
• effect 4.0.6
• prophet 1.4.0
• Full benchmark report

Flat growth

• Run 2026-11-07T181422-912Z-3a5a6cec on 2026-11-07, commit 3a5a6ce
• Both libraries ran in linux/arm64 containers on Apple M4 Pro
• effect-prophet 0.0.0
• effect 4.0.6
• prophet 1.4.0
• Full benchmark report

Logistic growth

• Run 2026-11-07T184211-397Z-3a5a6cec on 2026-11-07, commit 3a5a6ce
• Both libraries ran in linux/arm64 containers on Apple M4 Pro
• effect-prophet 0.0.0
• effect 4.0.6
• prophet 1.4.0
• Full benchmark report

Timings come from one machine and describe this workload only. They are not a general speed ranking. Where Python is faster, it is often because Python runs Stan's optimizer as compiled C++, while Effect Prophet runs the same algorithm as WebAssembly with narrower vector instructions. The gap is widest for Newton's method on short histories with many parameters.

Linear growth

Rolling-origin cross-validation, metrics, baselines, settings search and a final holdout check. All linear growth benchmarks →

Case	Input	Effect output	Python output	Largest difference	Timed step	Effect time	Python time
Cross-validation with a forecast regressor	256 rows	106.8125	106.8124	0.0025%	Cross-validate (different work)	51.6 ms	185 ms
Cross-validation with a forecast regressor and ranges	256 rows	106.8125	106.8124	0.0025%	Cross-validate with ranges (different work)	51.9 ms	255 ms
Cross-validation and metrics	96 rows	108.5823	108.5823	< 0.0001%	Cross-validate (different work)	10.1 ms	72.0 ms
Cross-validation with ranges	96 rows	108.5823	108.5823	< 0.0001%	Cross-validate with ranges (different work)	10.5 ms	78.2 ms
Choosing settings and a final holdout	96 rows	108.5823	108.5823	< 0.0001%	Search settings (different work)	20.2 ms	151 ms
Search with a failing candidate	96 rows	108.5823	108.5823	< 0.0001%	Search settings (different work)	21.4 ms	153 ms
Choosing settings by a custom metric	96 rows	108.5823	108.5823	< 0.0001%	Search settings (different work)	20.4 ms	143 ms
Twelve folds, three years, sequential and parallel	1095 rows	420.8894	420.8323	0.052%	Cross-validate (different work)	953 ms	2.61 s
Cross-validation with repeated timestamps	96 rows	15.5221	15.5221	< 0.0001%	Cross-validate (different work)	6.85 ms	53.4 ms

Flat growth

Rolling-origin cross-validation, metrics, baselines, settings search and a final holdout check on flat models. All flat growth benchmarks →

Case	Input	Effect output	Python output	Largest difference	Timed step	Effect time	Python time
Cross-validation with a forecast regressor	730 rows	45.0435	45.0356	0.094%	Cross-validate (different work)	60.3 ms	227 ms
Cross-validation with a forecast regressor and ranges	730 rows	45.0435	45.0356	0.094%	Cross-validate with ranges (different work)	61.8 ms	291 ms
Cross-validation and metrics	96 rows	24.0746	24.0746	< 0.0001%	Cross-validate (different work)	4.50 ms	46.2 ms
Cross-validation with ranges	96 rows	24.0746	24.0746	< 0.0001%	Cross-validate with ranges (different work)	5.57 ms	61.3 ms
Choosing settings and a final holdout	96 rows	24.0746	24.0746	< 0.0001%	Search settings (different work)	12.3 ms	146 ms
Cross-validation, eight months	256 rows	34.9930	34.9930	< 0.0001%	Cross-validate (different work)	6.27 ms	39.0 ms
Cross-validation with ranges, eight months	256 rows	34.9930	34.9930	< 0.0001%	Cross-validate with ranges (different work)	8.36 ms	49.1 ms
Twelve folds, sequential and parallel	256 rows	35.0008	35.0008	< 0.0001%	Cross-validate (different work)	27.6 ms	184 ms
Monthly, custom and scaled metrics	256 rows	49.8561	49.8561	< 0.0001%	—	—	—

Logistic growth

Rolling-origin cross-validation, metrics, baselines, settings search and a final holdout check on logistic models. All logistic growth benchmarks →

Case	Input	Effect output	Python output	Largest difference	Timed step	Effect time	Python time
Cross-validation and metrics	96 rows	59.1345	59.1345	< 0.0001%	Cross-validate (different work)	17.1 ms	59.5 ms
Cross-validation with ranges	96 rows	59.1345	59.1345	< 0.0001%	Cross-validate with ranges (different work)	18.1 ms	64.1 ms
Choosing settings and a final holdout	96 rows	59.1345	59.1345	< 0.0001%	Search settings (different work)	35.5 ms	115 ms
Cross-validation, eight months	256 rows	88.5367	88.5214	0.034%	Cross-validate (different work)	14.1 ms	41.5 ms
Cross-validation with ranges, eight months	256 rows	88.5367	88.5214	0.034%	Cross-validate with ranges (different work)	15.1 ms	56.4 ms
Twelve folds and a search, sequential and parallel	256 rows	90.7830	90.7874	0.079%	Cross-validate (different work)	61.3 ms	170 ms
Monthly, custom and scaled metrics	256 rows	90.9241	90.9228	0.20%	—	—	—