A Forecast We Could Not Take Back

Jason Duke, Founder, Kronaxis

Tag: Research

In April we built sixty five thousand synthetic people, had them project the English local elections across 136 councils and committed the hash of every projection to GitHub before a single vote was counted. That last part is the whole point. A forecast you can quietly edit after the result is not a forecast. This one could only be right or wrong, in public, against a future nobody had counted yet.

What it got right

On the national vote share, the panel of made up people landed at a mean absolute error of 2.96 points against the BBC projected national share. In the same cycle YouGov and Opinium came in at 3.2 and Electoral Calculus at 5.0. On that one number, a synthetic panel was the most accurate forecaster of the field. At ward level, a fifty person panel across ten by elections came in at 7.5 points against a statistical baseline of 8.8.

What it got wrong, stated plainly

Council control was a failure. We called the controlling party correctly 28 per cent of the time, and a rule as blunt as "the incumbent holds" scores 32 on the same councils. We lost to the simplest baseline there is. Most of the misses were councils that finished in no overall control, an outcome our pipeline never once produced. The answer options we gave the model did not span the answers reality chose.

And the caveat that belongs in the opening line, not a footnote: most of that national accuracy came from the make up of the pool, not from any clever reasoning. The party mix of the sixty five thousand already sat close to the answer before a single model step ran. In this build the field we scored against was, in effect, already written into each persona. The headline is real. It is also capped, and we can tell you exactly why.

It beat the pollsters on one number and lost to a coin on another. Both are true, both are public and the reasons for each are on the table.

Why publish the weakness next to the win

Because the next experiment is only obvious once you admit the confound: take the answer out of the persona and see whether the signal survives. That is the study we are now setting up with survey scientists at the University of Michigan. A forecast that is committed before the count, scored against a counted result and honest about where it leans on the pool rather than the reasoning, is worth more than a triumphant press release that nobody can check. The commitment, the projections and the counted results sit side by side, public, at kronaxis.co.uk/election-results, with the hash committed on GitHub before polls opened.

That is the discipline behind everything we make. Provable, not plausible. It applies to a synthetic electorate exactly as it applies to a company ownership graph: commit in advance, score against reality and publish the number that does not flatter you next to the one that does.