Accuracy is not how often you are right. It is whether your 70 percent happens 70 percent of the time.

The wrong measure

The intuitive measure of a forecaster is how often they are right. It is wrong, and replacing it is the first step in becoming better.

A forecaster who says 80 percent and is right is not more accurate than one who says 60 percent and is right, if the event was in fact a 60 percent proposition. The first was overconfident and happened to be lucky. The second was correct about the probability, which is the thing a forecast actually claims.

Hit rate rewards boldness. Say 95 on everything, be right most of the time, and look accurate. But the forecasts that fail will fail expensively, because they were sized for a certainty that did not exist.

Calibration

A forecaster is calibrated when, across all the occasions on which they said 70 percent, the event occurred about 70 percent of the time; across all the occasions they said 90, about 90; and so on down the scale.

Plot it and a calibrated forecaster sits on the diagonal. Above the line is underconfidence: things happened more often than stated. Below it is overconfidence, which is the more common failure and the more expensive one.

Calibration rewards honesty about uncertainty. Unlike hit rate, it is a skill that can be improved by practice, because it produces a score that can be compared across time. A forecaster can watch their own line move toward the diagonal, and know that it has.

Why the experts cannot do this

A commentator has a view. A forecaster has an estimate with a revision history.

The difference is procedure. An estimate is a number, written down before the price is seen, anchored to how often this kind of thing has happened before, adjusted for what is different this time, and then moved in proportion to each piece of evidence that arrives. Every one of those steps is skippable, and skipping any of them is invisible: nothing feels wrong, the forecaster simply never learns what they would have thought.

A commentator skips all of them, not through carelessness but because the format does not ask for any of them. There is no number to record, no base rate to anchor to, and no score at the end. The forecast is delivered, the outcome arrives, and the next forecast is invited regardless.

Scoring

A recorded forecast can be scored, and the standard score is simple enough to do by hand. Take the stated probability, subtract the outcome written as 1 or 0, and square the difference. A forecast of 70 percent on an event that occurred scores 0.09. The same forecast on an event that did not occur scores 0.49. Lower is better.

A forecaster who always says 50 percent scores exactly 0.25 on every forecast, so 0.25 is the line: below it a forecaster is adding information, above it they are subtracting it. And the score is honest in a specific sense. It is minimised by stating your true probability, so there is no advantage in hedging or exaggerating.

Over a few dozen forecasts, the average is a measure of the forecaster that no panel or column has ever produced about its author. It is the difference between claiming to be accurate and being able to show it.

The benchmark that already exists

A prediction market price is a forecast. It can be scored the same way, on the same questions, over the same period, and the result is a benchmark that keeps its own score continuously and for free.

Beating it means producing a lower score than the price across a meaningful number of resolved questions. That happens. It happens rarely, more often on obscure contracts than liquid ones, and more often to forecasters who have done the work than to those who have not.

It also states the risk plainly. A price of 70 that resolves against its holders three times in ten is not a failed benchmark. It is a correct one. Beating the market does not mean avoiding loss. It means losing less often than the price says, over time, and being able to prove it.