Forecast Evaluation · Kept
A walk-forward backtest of every season since 2022-23
After filling data gaps, every game from 2022-23 on was predicted by a model that had never seen that season, using that day's injury report and no betting lines.
Why a backtest
A few weeks of live forecasts cannot tell you much. Thirty games can show 70% or 55% winners just by chance. To learn how good the model really is, it has to be tested on thousands of games, and the only way to get that many is to go back in time. The danger in going back is leakage: using anything the model could not have known on the day of the game.
Data work first
Three gaps had to be closed before the backtest could be trusted:
- 209 days of 2022-23 official injury reports were missing. Without them the availability input would fall back to “who did not play”, which is only known after the game.
- 260 games had no box score, which breaks the rotation definitions and recent-form inputs.
- Team form had to go back to 2021 so the first backtest season had a full ten-game history.
Method
Each season is predicted by a model trained only on the seasons before it. Every game then goes through the same function the daily forecast uses, predict_single_game, with that day’s archived injury report:
def run_season(season, df, log):
import model.predict as P
from model.train import train_production_models
from data.ingestion.fetch_official_injuries import get_injury_report
train = df[df['season'] < season]
if train.empty:
log(f"{season}: no earlier seasons to train on — skipped")
return
train_through = train['season'].max()
log(f"{season}: training on {len(train)} games "
f"({train['game_date'].min().date()} -> {train['game_date'].max().date()})")
model_home, model_away, model_margin = train_production_models(train)
calib, n_cal = fit_calibration(season)
log(f"{season}: win-prob calibration = "
f"{'Platt on %d earlier wf rows' % n_cal if calib else 'fixed sigmoid'}")
P.get_player_predictions = no_player_model
done = done_game_ids(season)
final = final_game_ids(season)
dates = season_dates(season)
made = skipped = 0
for i, d in enumerate(dates):
games = [g for g in P.get_todays_games(game_date=d, include_final=True)
if g['game_id'] not in done and g['game_id'] in final
and g['season_type'] in ('Regular Season', 'Playoffs')]
if not games:
continue
preds = []
with contextlib.redirect_stdout(io.StringIO()):
injury_report = get_injury_report(d)
for g in games:
try:
pred = P.predict_single_game(
g, model_home, model_away, injury_report, {},
model_margin=model_margin, calibration_model=calib)
except Exception as e:
pred = None
err = f"{type(e).__name__}: {e}"
else:
err = "no features"
if pred:
preds.append(pred)
else:
skipped += 1
log(f" skip {d} {g['away_team']}@{g['home_team']}: {err}")
save_and_grade(preds, train_through)
made += len(preds)
if i % 10 == 0:
log(f" {season} {d}: {made} predicted, {skipped} skipped ({i + 1}/{len(dates)} dates)")
log(f"{season}: done — {made} predicted, {skipped} skipped")Win probabilities need the same care. The curve that turns a forecast margin into a probability is refit for each season, using only backtest forecasts from earlier seasons:
def fit_calibration(before_season):
"""Platt scaling on walk-forward predictions from seasons before the target."""
q = text("""
SELECT p.model_margin, (g.home_score > g.away_score)::int AS home_won
FROM predictions p JOIN games g USING (game_id)
WHERE p.model_version = :v AND g.season < :s AND g.is_final
""")
with engine.connect() as c:
rows = c.execute(q, {'v': WF_VERSION, 's': before_season}).fetchall()
if len(rows) < MIN_CALIBRATION_ROWS:
return None, len(rows)
X = np.array([[float(r[0])] for r in rows])
y = np.array([r[1] for r in rows])
return LogisticRegression().fit(X, y), len(rows)Results
The backtest now covers 5,390 games from 2022-10-18 to 2026-11-08. It picked 65.1% of winners. Always picking the home team would have picked 55.6%.
Winners picked by season: model vs always picking the home team
Data table
| Model | Home team | Difference | Games | |
|---|---|---|---|---|
| 2022-23 | 62 | 58.1 | 3.9 | 1,314 |
| 2023-24 | 64.3 | 54.6 | 9.7 | 1,312 |
| 2024-25 | 66.7 | 54.6 | 12.1 | 1,309 |
| 2025-26 | 67.2 | 55.5 | 11.7 | 1,309 |
| 2026-27* | 67.1 | 51.4 | 15.7 | 146 |
Accuracy rises from 2022-23 to 2025-26, partly because each later season has more history to train on. Full detail, including calibration and margin errors, is on the Performance page.
Caveats
- These are retrospective tests, not forecasts published at the time. The site always keeps them separate from issued forecasts.
- Ten games where the source table stores the same team as home and away are left out of the published figures; see Methodology.