Which forecast should you trust?

The question

This is a solo, self-directed study on public Citi Bike trip data, built over several weeks as a portfolio case study rather than employer work.

Demand forecasts get consumed at different grains by different decisions: tomorrow's rebalancing plan needs station-day numbers, a quarterly ops review needs region-month, and capacity planning needs station-year. The standard bake-off — one metric, one horizon, one winner — answers none of those directly. This is a designed comparison built to answer the question teams actually face: which class of model should own which decision?

To make the error metric concrete: WAPE of 23% means that for every 100 rides that happened, the forecast misallocated 23 rides' worth. At a busy 200-ride/day station, the gap between the 23% champion and a 30% baseline is roughly 14 bikes a day of misallocated truck capacity — every day, at each of the hundreds of stations that size.

The contestants, spanning paradigms on purpose. From classical statistics: per-station ARIMA (a 1970s workhorse that projects the recent past forward), plus AutoETS, AutoARIMA, MSTL, and a harmonic regression with weather inputs. From machine learning: XGBoost, a gradient-boosted tree model — "the tree" from here on — trained on all 1,481 stations at once, with calendar and weather features, in several variants so features can be added and removed cleanly. From the AI shelf: Chronos-Bolt and Chronos-2, foundation models — neural networks pre-trained by Amazon on millions of unrelated time series (sales, traffic, electricity), used here zero-shot: they see the bike data only at prediction time and are never trained on it. And two deliberately dumb baselines: "same as last week," and the last-year lookup — the average of the same weekday one year earlier. Every model predicts every station at horizons from 1 to 90 days.

Evaluation design (where these comparisons usually go wrong)

• One predictions table — every model × 1,481 stations × the same eval year (Jun 2025–May 2026), so any aggregation (station-day, region-day, system-day, station-year) is a groupby, not a rerun.
• Horizon sweep at 1, 7, 14, 28, 56, and 90 days ahead, with forecast start dates rotated every 8 days so no weekday is favored; all models scored on matched forecast/target pairs.
• No time travel: every prediction uses only data available before the day being predicted — models are never refit on future data, and lag features are shifted to match the horizon. The one deliberate exception is weather: models marked † get the target day's actual weather, which is a fair stand-in for a forecast at 1 day ahead and impossible knowledge at 90. So every weather model also runs clean (without it), at every horizon, and long-range conclusions are drawn from the clean runs.
• Cost ledger: wall-clock time, CPU, and memory recorded per model, because "0.5 points more accurate" is not a conclusion until you know it cost 40× the compute.
• Uncertainty on every headline gap: bootstrap confidence intervals (2,000 draws), resampled at the level where errors actually correlate — whole days or whole forecast starts, never station-days, since one rainstorm moves 1,400 stations together and resampling station-days would fake precision. Two apparent wins got demoted to ties by this step; they're reported as ties.

Results

1. The best model for tomorrow is the worst at the annual total

Predicting tomorrow, station by station, the tree wins: 18.4% error versus 20.3% for ARIMA and 30.0% for the trailing average — and its edge is biggest at high-traffic stations, where the trips actually are. But add up each model's 365 daily forecasts into a yearly total per station, and the standings invert. ARIMA's totals land within 0.2% of reality at the median station; the tree's are off by 1.2% — six times the drift, and consistently low at growing stations. The reason is mechanical, not mysterious: the tree forecasts tomorrow as a ratio to the station's recent average, which is structurally a bet that next month looks like last month, so where ridership is growing it always lags reality. ARIMA projects the trend forward by construction, so its errors cancel over a year instead of accumulating. Decision consequence: the model that plans tomorrow's trucks should not sign off on next year's dock purchases — the best short-range shot and the best long-range bookkeeper are different models.

2. Models that lean on the recent past all fail the same way, at the same range

ARIMA, anchored to the latest data, wins at 1 day ahead; by 7 days the tree leads, and the gap widens from there. By 90 days, every model whose only idea is "the future resembles the recent past" collapses into the same failure — ARIMA at 62.6% error, the trailing average at 65.5% — because a forecast made in February simply doesn't know what May is like. Give a model a picture of the annual cycle (MSTL, which explicitly decomposes the year's shape) and the 90-day error drops to 35.5%, at some cost to short-range sharpness. Decision consequence: assign model ownership by horizon — and the handoff point is measurable, at about a week.

3. At long range, nothing honest beat "check what happened last year"

The last-year lookup — no model, just the average of the same weekday one year earlier — scores roughly 29–30% error at every horizon, by construction: its accuracy doesn't depend on how far ahead you ask. At 90 days that flat line beats every clean model in the field by a statistically significant margin (5.7 points better than the tree [2.4, 9.6], 4.2 better than the AI model [0.7, 7.9]). Across the entire study, no entrant restricted to honestly available information ever significantly beat the lookup at ninety days — five purpose-built challengers tried. Catching this was the most important result in the study: this baseline was nearly left out of the field, and every long-range claim would have been inflated without it. Decision consequence: past about six weeks, spend the complexity budget on monitoring, not modeling.

4. The weather data earns its keep on 58 days of the year

Retrain the tree without its weather inputs and the average barely moves — about 1 point worse on ordinary days. On rainy days it's 13 points worse. Averaged feature-importance scores hide this completely: the weather data is insurance for the hard days, not a general boost. The ablation also settles an "AI vs. classical" talking point: without weather, the tree doesn't just tie the 1970s model, it loses to it (by a small but statistically real 0.7 points). The tree's headline advantage was never the algorithm — it was that one contestant got to read the weather and the other didn't. One more check that the weather value is real behavior and not model artifact: a separate analysis of the same system, built two years earlier with different methods, measured which stations' riders are most weather-sensitive — and the stations where weather features help the model most are the same stations (correlation +0.54). Two independent methods drew the same map.

The station-level version is live as an interactive weather explorer: filter by temperature, rain, time of day, and day type to see where demand moves most.

The reverse experiment makes the point from the other side: give a per-station classical model the same weather inputs the tree gets, and at 1 day ahead it comes within 1.4 points of the tree. At 90 days, though, it trails by 19 points — what remains, once the information is equalized, is the value of the tree learning all 1,481 stations at once instead of one at a time. Decision consequence: at short range, buy better data before a better algorithm; at long range, pooling across stations is the architectural choice that pays.

5. Why does an AI model that has never seen a bicycle do so well? We tested it.

Chronos-Bolt — used zero-shot, no training on bike data — significantly beats every trained model at 28–56 days ahead and ties them at 90, after three minutes on a consumer GPU. Suspicious. Our hypothesis: the model is handed the last 512 days of each station's history as input, that window contains last spring, and the transformer is smart enough to look it up — retrieval, not foresight. The test: shorten its input window to 256 days, so last year falls out of view. Its 90-day error collapses from 33.6% to 48.6% while its 1-day error doesn't move. The advantage was the memory, and the experiment — not a plausible story — is what says so.

6. The explanation made a prediction, and the prediction held

If the AI model's edge is really "it can see last year," the edge should be erasable: hand the tree the same information — last year's counts as three extra input columns — and the gap should close. It did, exactly: the AI model's significant mid-range lead over the clean tree became a statistical tie (−1.2 points [−3.8, +1.5]), and the upgraded tree noses ahead at 90 days. Two side results came free. The tree with weather and last-year columns became the strongest single model at every horizon — for 71 seconds of training. And the humbling one: even that tree, given the lookup's own numbers as inputs, still could not significantly beat the raw lookup at 90 days (+4.2 points [−0.1, +9.7]).

7. Equal information, equal accuracy — the study's tidiest result

Chronos-2, the newer and smaller foundation model, can accept side information — so it enables the cleanest experiment in the study. Fed the same weather inputs the best tree gets (still zero training on bike data), it ties that tree at every single horizon — every difference within the error bars. A model trained on 1.8 million station-days of this system and a model that has never seen a bicycle, given the same information, are the same forecaster. Its input window told the same story as before: at the same 512-day window it matches its older sibling (architecture upgrade bought nothing), and only with 1,024 days — two winters to average instead of one — does it set the zero-shot record. Decision consequence: for a new deployment with no training pipeline, a zero-shot foundation model plus a data feed reaches parity with a tuned in-house model — build-vs-buy is now an empirical question, and here it's a wash on accuracy and a win on time-to-first-forecast.

8. The overall winner is the arithmetic mean

Average the best tree's forecast with the AI model's forecast, fifty-fifty — two columns that already existed, zero training — and the result beats everything, at every horizon, including both of its parents (at 90 days, +1.9 points versus the better parent [0.5, 3.2]; significantly so from two weeks out). The control experiment explains why: blending the two clean models only ties them. Averaging pays when the parents make different mistakes — the weather-fed pair err differently because they use their shared information differently, while the clean pair lean on the same last-year signal and average to nothing new. One more zero-training experiment closed the loop: adjusting the last-year lookup for how busy this year has been helps at 1 day (+3.1 points) and hurts at 90 (−3.9) — the lookup wins long-range because it ignores the recent past, not despite it. Decision consequence: before buying any new model, average the two you have — it was the cheapest accuracy in the entire study.

Explore the race yourself

Every entrant, every horizon, with uncertainty bands — toggle models on and off, or switch between the story view and the final standings:

(Or open the full-page version.)

What I'd take to production

• Day-ahead rebalancing: the tree with weather and last-year inputs (result 6), with conformalized quantile bands for the service-level decision. In the companion study, raw 80% bands covered only 64% of outcomes; conformalization restored coverage to 79.5%. Blend with Chronos-2 where the inference budget allows (result 8).
• Weekly-to-monthly planning: same model, weather dropped beyond the range real forecasts can cover.
• Anything past ~6 weeks: the last-year lookup — the best trained model only ties it there, and the lookup is auditable by anyone in the room — with the saved complexity budget spent on monitoring.
• Annual capacity: bias-audited per class of station; never the raw tree output at growing stations (result 1).
• Monitoring and feedback: retrain the tree monthly (71 seconds makes the cadence free); track rolling 28-day error against the lookup as a drift alarm — the lookup is the natural fallback precisely because it cannot drift; and re-run the annual-bias audit quarterly, since result 1 says daily accuracy will not warn you when annual bias appears.

What this case study demonstrates

Evaluation design under leakage pressure (rolling forecast starts, horizon-shifted features, impossible knowledge flagged and quarantined); uncertainty discipline (bootstrap intervals sized to the real correlation structure, with two headline claims demoted to ties and reported as such); testing explanations by experiment rather than narration (the input-window ablations, the predicted falsification in result 6); cost-aware model selection (a wall-clock ledger beside every accuracy claim); and production judgment — including the willingness to recommend a zero-parameter lookup over my own best model where the evidence says so.

Limitations

Actual weather stands in for weather forecasts, so reported skill is an upper bound (archived NWS forecasts are the natural next experiment). Observed trips are demand censored by supply: an empty dock records zero departures no matter how many riders walked away. The eval year is one draw of NYC weather and one network regime. And the two strongest entrants (the upgraded tree and the blend) were designed after other results were known — I'd confirm both on a fresh test year before staking a real decision on them.

Stack: pandas/pyarrow pipelines, statsmodels & statsforecast, XGBoost, Chronos on Intel Arc via torch-xpu. ~540k-row reproducible predictions table.