Grid & Gradient
Model Audit — August 2026

Grid&Gradient

A full audit of FormulaAnalytics — twelve automated workflows keeping a Kalshi trading operation running, and a hand-derived gradient hiding inside the win-probability engine that decides what to trade. Read top to bottom, or jump straight to the math.

12GitHub Actions workflows
20,000Monte Carlo sims / prediction
3regulation eras tracked
0.07 → 0.10σ that fixed a ranking bug
0containers, databases
01

FormulaAnalytics Stack

F1 win-probability model + Kalshi prediction-market trading

A pure Python quant project — no containers, no traditional database, no JavaScript build tooling. The interesting engineering isn't the language, it's the automation wrapped around it: twelve scheduled pipelines, two Cloudflare Workers, and a Windows machine that wakes itself up to catch live telemetry.

Core dependencies

  • Data / ML — pandas, numpy, scipy, fastf1 (telemetry & timing), pyarrow, scikit-learn
  • Scraping / parsing — beautifulsoup4, pymupdf, RapidFuzz
  • Storage — boto3 against Cloudflare R2 (S3-compatible object store) — no SQL, no ORM
  • Ops — cryptography (RSA-PSS session signing), apprise (push/SMS alerts), anthropic (LLM extraction & agent reasoning — deliberately kept out of the pricing path)
  • Dev — pytest, ruff, mypy, all configured inline in pyproject.toml; package installed editable

CI / CD — 12 GitHub Actions workflows

WorkflowTriggerPurpose
ci.ymlEvery push · PR→mainruff + mypy on every push; full pytest suite gates merges into main
f1-ops.yml~16 cron schedulesRace-weekend lifecycle: telemetry, snapshot, predict, record, alert, settle, retro
market-ops.ymlReusable / callableShared entry point every market's ops workflow runs through
model_gate.ymlWeekly, Mon 08:00 UTCReal-data walk-forward validation + Kalshi backtest + LLM gate digest
conductor.ymlrepository_dispatch onlyLLM-agent orchestrator; deliberately not on GitHub cron (drift-prone)
conductor_clock_deploy.ymlPush to infra/ · manualDeploys the Cloudflare Worker that heartbeats the Conductor every 15 min
weekend_dashboards.ymlCron, race weekendsRenders projection dashboards, deploys to a private Cloudflare Worker site
backfill_odds.ymlWeekly + manualReconstructs full Kalshi historical odds from candlesticks
backfill_results.ymlManual onlyFull-history race-results backfill from the Jolpica API
base_evolution.ymlWeekly, Tue 10:00 UTCLLM-validated changelog process, appends to an audit trail
stewards_feed.ymlHourly (race weekends)LLM extraction of FIA steward decisions; custom git merge driver for concurrent commits
x_context_dryrun.ymlManual onlyInspection tool for social/qualitative LLM research agents

Infrastructure

Hosting
2× Cloudflare Workers (dashboard site + conductor heartbeat)
Storage
Cloudflare R2, flat CSV/JSONL for seed data
Markets
Kalshi API (trading), Jolpica API (race results)
Local automation
Windows Task Scheduler wakes the PC for live telemetry pulls
Containers
None — no Dockerfile anywhere in the repo
Database
None — flat files + object storage only
02

How The Model Thinks

Eight subsystems, one race prediction

The win-probability engine is a Plackett-Luce race model — full derivation in Section 03 below. Everything else on this page is what surrounds it: a separate reliability model for DNFs, a separate model for qualifying, a Monte Carlo simulator that turns win probabilities into a full outcome distribution, and a trading layer that decides what to do about the gap between the model and the market.

DNF / reliability model

Not logistic regression — an empirical shrunk hazard-rate model with two independent families, because a DNF is either a car problem or a driver problem and they decay at different speeds:

  • Mechanical hazard — per-team, 30-race half-life (car reliability shifts fast between seasons), shrunk toward the field base rate with pseudo-count k=40
  • Accident hazard — per-driver, 80-race half-life ("trait-like" incident-proneness), shrunk with k=80; scaled further by a grid-position bucket multiplier (lap-1 risk, buckets 1–5 / 6–12 / 13+, k=150) and a street-circuit multiplier (clipped 0.5–3.0, k=300)

Combined as p_dnf = 1 − (1−p_mech)(1−p_acc), capped at 0.60. A separate lore-derived multiplier (×1.20) bumps mechanical hazard for drivers on a fresh power-unit element beyond their season allocation.

Qualifying model

A structurally separate Plackett-Luce fit on Saturday's order, not Sunday's — "one-lap pace rewards different traits than race pace: peak downforce and tire warm-up, not tire management." Grid distributions (not point estimates) come from sampling Gumbel noise into the fitted strengths and re-ranking — proper Plackett-Luce sampling, 2,000 samples per live prediction.

The race model's anchoring fix (Section 03) was tried here too and made every metric worse on the 2026-only split — mean Spearman, top-3 hit rate, pole log-loss, and expected-grid MAE all degraded monotonically with anchor strength — so it stays off for qualifying specifically.

Ensemble research only

A scikit-learn HistGradientBoostingClassifier (depth 3, 300 iterations, early stopping) exists and combines with the Plackett-Luce output via a log-linear pool — a product-of-experts blend, weights fit walk-forward by minimizing log-loss. But grepping the live prediction path found fit_gbm / predict_gbm_race have no callers anywhere in production. It's a backtest module the live pipeline never calls — win probabilities ship from Plackett-Luce + the simulator + the hazard model, no gradient boosting involved.

Race simulator

A vectorized Monte Carlo engine, 20,000 simulations per live prediction (4,000 inside the backtest loop, for speed). Four layered mechanisms, all vectorized across the simulation axis:

MechanismDraw
Race-day pacetheta_pace + Normal(0, 1.283·τ)
Lap-1 shufflegrid slot + Normal(0, 1.5), re-ranked
DNFsmechanical ~ Bernoulli(p_mech); accident lap ~ Beta(0.7, 2.5) — front-loaded
Safety carsn_sc ~ Poisson(0.15 × track.sc_likelihood)

Survivors are ranked by score = pace − κ·log(pos1) + sc_shock, where κ inflates the fitted grid weight ×2.0 to restore the grid's full race-long persistence — the single-race fit dilutes that weight against the pace covariate, but a whole race needs it undiluted (calibrated against real P(finish≤k | grid) curves, cutting RMSE from 0.066 to 0.050). A wet-weather branch (Bernoulli per draw) compresses pace spread 0.65×, doubles safety-car rate, and multiplies accident hazard 1.8× — with rain_prob=0 byte-identical to the dry path. For pre-qualifying predictions, each of the 2,000 sampled grids from the qualifying model contributes 100 race draws, pooling Saturday's uncertainty directly into Sunday's distribution.

Calibration

Two layers, both fit walk-forward (prior seasons only, applied to the current one): temperature scaling on the win-probability softmax (fits τ to minimize out-of-sample winner log-loss), and Platt scaling for binary contract probabilities like top-10 finishes. Metrics: winner log-loss, multiclass Brier, and a midfield-band report restricted to p∈[0.15, 0.75] — "winner log-loss is favorite-dominated; the monetizable edge lives in the midfield." Significance is a paired bootstrap, 10,000 resamples, reporting a 95% CI and P(better).

Trading layer

Fee model
⌈0.07 · price · (1−price)⌉
Edge threshold
Model edge must clear fees + 2pp buffer
Sizing
Quarter-Kelly, capped at 5% bankroll / market
Per-race cap
15% bankroll across correlated legs
Open exposure cap
30% bankroll across all events
Kill switch
25% season loss halts live trading until a human re-enables it

A divergence scanner runs two families: structural — containment violations like the winner price exceeding the podium price, priced net of both legs' fees — and model-based, where the simulator's joint outcome distribution prices every market type consistently, so any market's touch price vs. the simulator becomes a candidate trade. Slippage is modeled from order-book depth, currently "honestly empty" since recorded books show zero resting orders — live sizing stays at the liquidity-cap floor until that changes.

Trackfit — a feature that shipped twice

v1 and v2 projected hand-typed car/driver attribute ratings onto per-circuit "track demand" vectors. Both cleared winner log-loss narrowly — and both significantly hurt top-10 log-loss, with the effect direction actually inverting in the live-relevant window. Both were rejected; the adoption question was explicitly closed.

v3 replaced the hand-typed vectors with ones measured from telemetry — same mechanism, better inputs — and every metric cell improved, including on held-out data. It shipped. But it's gated to contribute exactly zero signal for the first 12 rounds of a new regulation era, so despite being "on" by default, it's functionally inert for most of the 2026 season — real magnitude only starts around round 13, once 2026-era ratings have enough data to clear the gate.

03

Inside Plackett-Luce

The exact math, line by line, from pl_model.py

Every driver and every team gets a strength score. Every finishing order is a sequence of softmaxes over who's still in contention. The rest of this section is what that sentence means in code.

The objective function

Linear predictor per driver i in race r, era-partitioned so a driver switching teams doesn't import the old team's pace into his own rating:

theta — strength score θri = α[driver(i), era(r)] + β[team(i), era(r)] + Xri · w

The race-level Plackett-Luce likelihood — the observed finishing order is a sequence of softmaxes, one per finishing position, over whoever is still left in the field:

per-race log-likelihood loglikr = Σk [ θ(k) − logsumexp( θ(k:n) ) ]

Summed across every race with a recency weight — 12-race exponential half-life, plus an extra 0.5× cut for any race from a different regulation era than the one being predicted:

race weight wr = 0.5 ^ ( (t_predict − tr) / 12 ) × ( 0.5 if era(r) ≠ era(target) else 1 )
full objective — minimized via L-BFGS-B Objective(α,β,w) = −Σr wr · loglikr
+ ½Σ(α−αanchor)² / σ_α²
+ ½Σ(β−βanchor)² / σ_β²
+ ½Σ((w−wanchor) / σ_w)²

This is regularized MLE, not full Bayesian inference — the L2 terms act as Gaussian priors, making it a MAP estimate. Two things worth noticing: DNFs are excluded from the likelihood entirely rather than ranked last (a retirement carries no pace signal — reliability is its own model, above), and the anchors αanchor / βanchor aren't fixed constants — they're the fitted output of a second, grid-free Plackett-Luce model, which is the entire subject of Section 03.6.

The gradient trick

No autodiff — the gradient is hand-derived and passed to SciPy as jac=True. The suffix logsumexp needed at every finishing position is computed backward through the field in O(n) via running logaddexp accumulation, not by recomputing logsumexp separately at each position:

suffix logsumexp — O(n), overflow-saferunning = −∞ for k from n−1 down to 0: running = logaddexp(running, θ[k]) lse[k] = running

The per-position derivative needs, for every driver j, the sum of that driver's softmax weight at every stage up to and including its own finish — this is the piece the code comments call "the O(n²) part," built as a masked n×n matrix instead of a loop:

gradient — masked n×n, numerically safe by constructionexpo = θ[j] − lse[k] // (j,k) pairs mask = k ≤ j // lower triangle g[j] = 1 − Σ_k exp(expo) · mask

Inside the masked region k≤j, lse[k] is always ≥ θ[j] by construction (it's a logsumexp over a set containing j), so expo can never exceed zero there and the exponential can't overflow — the mask isn't just selecting the right terms, it's what makes the exponentiation safe. Correctness is pinned by two tests that compare this analytic gradient against SciPy's finite-difference approximation, with and without the anchor term active.

Every constant in the fit

All 18 PL-model constants, config.py
ConstantValuePurpose
HISTORY_START_SEASON2018Earliest season pulled into training history
REGULATION_ERAS[18–21, 22–25, 26–30]Season ranges defining the three era indices
DECAY_HALF_LIFE_RACES12Exponential recency-weight half-life, in races
CROSS_ERA_WEIGHT0.5Extra downweight on training races from a prior era
PRIOR_SIGMA_DRIVER0.5Unanchored driver-skill L2 prior std (shrink to 0)
PRIOR_SIGMA_CONSTRUCTOR1.0Unanchored constructor-pace L2 prior std
DRIVER_ANCHOR_FROM_RESULTSTrueShrink driver skill toward a grid-free fit instead of 0
PRIOR_SIGMA_DRIVER_ANCHORED0.07Tight driver prior std used only when anchored
CONSTRUCTOR_ANCHOR_FROM_RESULTSTrueShrink constructor pace toward the grid-free fit
PRIOR_SIGMA_CONSTRUCTOR_ANCHORED0.10Tight constructor prior std used only when anchored
CONSTRUCTOR_ANCHOR_SIGMA_ALPHA0.1Crushes driver skill inside the anchor sub-fit, so its beta reads as pure team rating
QUALI_CONSTRUCTOR_ANCHOR_FROM_RESULTSFalseSame anchor idea, decoupled for qualifying — off, it measured worse there
PRIOR_SIGMA_QUALI_CONSTRUCTOR_ANCHORED0.5Only used if the quali anchor flag above is re-enabled
EXCLUDE_DNF_FROM_PACETrueCensors non-finishers out of the ranking likelihood entirely
PRIOR_SIGMA_COVARIATE10.0Default loose L2 prior std on any covariate without its own sigma
SURVIVORSHIP_FINISH_RATE_FLOOR0.70Trading guardrail — drops BUY signals on teams below this finish rate
SIM_SIGMA_RACE_TAU_SCALE1.283Simulator pace noise = this × fitted temperature τ (≈ τ·π/√6)
SIM_KAPPA_SCALE2.0Inflates the simulator's grid weight to restore full race-long stickiness

Covariate registry

CovariateWhat it computesStatus
grid−log(grid position); pit-lane starts remapped to 21live — race
trackfitKalman-filtered car/driver attribute ratings × measured per-circuit demandlive — era-gated
fp_formWeekend long-run (race-pace) practice form, session-weightedlive — race
fp_quali_formOne-lap practice pace, raw session-weighted columnoff — gated
practice_form (quali)Precision-weighted blend of practice pace into qualifyinglive — adopted
practice_form_raceSame builder, registered for the race channel insteadrejected

practice_form_race is the one genuinely permanent no — its ablation row scored worse than doing nothing at all (1.1911 → 1.2020 winner log-loss, "the worst row in the whole matrix"), reasoned to be because the grid covariate already absorbs that pace information for the race model in a way it structurally can't for qualifying, which is fit before a grid even exists.

What the tests actually pin down

13 tests, tests/test_pl_model.py
TestAsserts
parameter_recovery_orders_drivers_correctlyRecovers a known driver ranking from simulated PL data (Spearman > 0.9)
grid_covariate_weight_recoveredRecovers the true grid coefficient sign and magnitude from synthetic data
gradient_matches_finite_differencesAnalytic gradient matches SciPy's numerical finite-difference gradient
gradient_matches_finite_differences_with_anchorsSame check, with the anchor-deviation term active
alpha_anchor_shrinks_toward_anchor_not_zeroAnchored fit pulls toward the anchor value, not toward zero
results_anchor_decomposes_grid_free_thetaAnchor's α + β exactly reconstructs the grid-free model's θ
grid_laundering_regression — driverThe literal Antonelli bug, pinned so it can never silently return
grid_laundering_regression — constructorThe literal Mercedes bug, pinned the same way
win_probabilities_sum_to_oneSoftmax output sums to 1 and ranks monotonically with θ
training_frame_after_cutoff_rejectedRefuses to train on any race at or after the prediction target — walk-forward integrity
time_decay_favours_recent_formA shorter half-life weights a recent strength jump more heavily
team_switch_does_not_leak_old_team_strengthThe era-partitioning regression test — a switcher's new team rating stays clean
temperature_calibration_recovers_overconfidenceThe temperature fit correctly detects and flattens an overconfident model

The anchoring saga

Era-partitioning itself is old, settled architecture. The anchoring mechanism above it is not — it was built, broken, and rebuilt in a single day, and the reason why is the best evidence for how this whole codebase operates.

It started with a driver in the championship lead fitting to the lowest skill rating of anyone on the grid.

Antonelli had won 5 of 9 races and led the championship. The unregularized model fit his driver skill (α) to −0.675 — dead last of 22 drivers. The mechanism: a driver who wins from pole every time gives the grid covariate first claim on the credit, and once grid alone explains the win, nothing is left over for α. Strip the grid covariate out entirely and his α becomes −0.007 — completely ordinary. That gap is the bug.

The fix was a second, grid-free Plackett-Luce fit whose output becomes the anchor — the L2 prior's mean — for the main model's α and β, with the prior tightened until it actually binds. Both anchor strengths were chosen by walk-forward sweep against real gate metrics, not picked by feel:

Driver-anchor σmodel LLsim LLtop10 LL
0.50 (loose)1.20781.26300.4845
0.101.19691.25380.4831
0.07 — chosen1.19651.25360.4830
0.031.19621.25380.4831

All three metrics improve monotonically as σ tightens, then go flat across 0.03–0.10 — 0.07 was picked from inside that flat plateau rather than the tightest value tested, for the same reason the simulator's noise constant (below) settled on the less-extreme side of its own optimum. Live effect on Spa, round 10: Antonelli's pre-quali win probability moved from 0.040 (lowest of the contenders) to 0.130 — into the leading pack alongside Hamilton and Russell.

The same session, checking the model against the market surfaced a second instance of the identical mechanism: Mercedes had gone 7-for-9 with 9 poles out of 9, and the model's team-pace rating for them fit to +0.054 — below Haas. The grid-free anchor for the same team said +0.767. A 0.71-unit gap, the largest of any team by a wide margin.

Constructor-anchor σmodel LLsim LLDNF LL
0.501.19651.25361.1971
0.151.19291.24121.1943
0.10 — chosen1.19141.24091.1929
0.031.18991.24241.1915

This one isn't clean-monotonic — winner and DNF log-loss keep improving all the way to σ=0.03, but simulator log-loss has a genuine interior optimum at 0.10–0.15 and gets worse below it. Traced per-race, the degradation concentrated entirely in the two 2026 races Mercedes didn't win — an overconfidence/tail-risk tradeoff, not noise. 0.10 was chosen because it's where the more reliable, 101-race simulator metric actually bottoms out. Result: Mercedes' team rating moved from +0.054 to +0.709, back in line with the other nine teams and correctly ranked first.

Independent second pass, same day

A separate audit re-derived both mechanisms from the code rather than trusting the narrative, and found one real gap: the qualifying model's anchor was silently sharing the race model's σ, despite the two never being validated together and the qualifying model not even using a grid covariate for its anchor's original justification to apply. A dedicated harness showed all four qualifying metrics got monotonically worse as anchor strength increased there. Fixed with a fully decoupled qualifying-side flag, defaulted off. Verdict: "Ship, with one real gap found and fixed."

The ripple continued downstream. The simulator's race-day noise constant had earlier been tuned down from its theoretical value (1.283) to 0.85 — which, after the anchor fixes landed, turned out to have been silently compensating for the anchor bug's inflated spread the whole time. A fresh resweep inverted the old table and restored 1.283.

Lessons written back into STATE.md
01A covariate correlated with a latent's target launders signal out of every latent sharing it — fixing β alone left α exposed to the identical mechanism.
02A prior can exist and still be functionally inert. "The anchor exists" is not the same claim as "the anchor binds" — sweep the strength against the real gate metric.
03A fix motivated by one specific case needs a leave-that-case-out check before the aggregate number can be trusted.
04Not every "tighter is better" sweep generalizes — check for interior optima, not just the trend line.
05A hyperparameter tuned while a structural bug was live may just be encoding compensation for that bug — re-sweep every downstream constant after fixing the structure underneath it.