What calibration means — and why a good 30% forecast usually doesn't come true
Say the drawdown risk on your screen reads 30%. The next day, nothing happens. Was the forecast wrong? If your instinct says yes, this article is for you — because judging a probability forecast is a skill of its own, and calibration is what makes a probability judgeable at all.
Probabilities that mean what they say
A model is calibrated when its probabilities mean what they say: when it says 30%, those events happen about 30% of the time. Not "the model feels 30% confident" — measurably, across many forecasts, the 30% bucket comes true about three times in ten. w4rn's forecasts are Platt-calibrated (a standard statistical correction, fitted out-of-sample) so the numbers are trustworthy, not just rank-ordered.
Calibration is not the same as skill
Two different questions, two different numbers. AUC measures ranking: does the model score genuinely risky days above calm ones (0.50 = coin flip and 1.00 = perfect)? Calibration measures honesty of scale: when that ranking is turned into a probability, is the probability right on average? A model can rank brilliantly and still exaggerate — shouting 80% for events that happen half the time. You need both, so we publish both: the AUC next to every forecast, and the calibration error (ECE) in the forecast's details — lower means the stated % tracks reality closer.
Why a good 30% forecast usually doesn't come true
Here's the trap most people fall into: a single outcome can never prove a probability right or wrong. If a calibrated model says 30% and the drop doesn't come, that isn't a miss — a correct 30% is supposed to fail seven times out of ten. A weather forecaster who says "30% chance of rain" isn't wrong on a dry day. The only fair test is the long run: gather every forecast that said "about 30%" and check that roughly three in ten saw the event. Judging one forecast by one outcome punishes exactly the honesty you should want from a risk tool — it rewards models that only ever whisper 1% or shout 99%.
How to read a calibrated number
Take a calibrated 30% for what it is: elevated odds, not a prediction. Compare it to that asset's own base rate, shown next to every forecast — 30% means very different things for a coin that drops sharply one day in twenty and one that does so one day in seven. Then act on the odds, not the outcome: elevated drawdown risk or a rising volatility outlook is a cue to size down and brace, whatever tomorrow happens to bring.
See today's live read in the cockpit → Nothing here is financial advice.