Outweigh helps a team compare options, see the evidence and assumptions behind each, and hear objections from clearly labelled synthetic customer perspectives. It keeps a sealed record of what was expected and why, so the decision can be reviewed against what actually happened. It does not forecast election results, share prices or exact future percentages, and we do not claim it can.
This page previously published error figures from two backtests: 48 public-survey questions and 19 commercial outcomes. On review we could not show those tests were blind. The answering model’s knowledge cutoff was assumed rather than confirmed by its vendor, many questions were published only weeks after that assumed date, and several came from yearly surveys whose earlier results the model may already have known. The same model answering without any of our process also scored closer on the 48 questions. A figure we cannot stand over does not belong here, so we have taken it down rather than caveat it. The runs are retained internally, unchanged.
Every case in the set has a published, unambiguous outcome that existed before the council was assembled. Nothing is scored against a judgement call.
Grounding is restricted to statistics published on or before the case's evidence cutoff, so the council cannot see the future through the source data.
The prediction is written to an immutable decision snapshot, timestamped, before the observed value is entered. The freeze is what makes the score meaningful.
The case list is fixed in advance and published in full, including cases the platform gets wrong. A benchmark you can edit after the fact is marketing, not calibration.
A synthetic panel is not valid everywhere, and a vendor who tells you otherwise has not measured it. Our position on each regime is written here before you have to ask.
Population-level figures for Ireland and Britain, anchored to official statistics, with a published error from a sealed single-pass run. This is the regime our numbers cover and the regime the published critiques find acceptable.
Read the direction, not the magnitude. Published work shows LLM panels exaggerate the distance between demographic groups, so a targeting call should be confirmed with human evidence before budget moves behind it. Our own measurement of this is in progress and will be published here whatever it shows.
No synthetic panel — ours included — beats a simple statistical baseline at the level of one named person. Where no official statistic resolves for a question, the run is labelled unanchored and is structured judgement, not an estimate.
Chen, Zhu and Zheng benchmarked simulated respondents against the General Social Survey and the World Values Survey across four models. Two findings matter: no model beat a simple statistical baseline at the level of an individual, and every model over-determined demographics — inflating the gaps between segments two to fourfold, pointing at the wrong segment in half of US cases, and manufacturing splits that do not exist in real people. Model size did not fix it.
Every figure above is a population aggregate — the regime that paper finds acceptable. It says nothing about gaps between segments, so we are running that test on ourselves: cases where the publisher reports the same question broken down by age, region, income or tenure, scored on gap inflation, whether we would pick the right segment, and how often we invent a split that is not there. Alongside it, a stats-only arm answers the same cases from the grounding layer with no persona reasoning, so the council has a baseline to beat.
Gap inflation (median)
0.88×
Right segment chosen
90%
Manufactured splits
0%
Result published here whatever it shows. Until it exists, treat any segment-level difference in a Outweigh report as directional and confirm a targeting decision with human evidence. Read the paper
The paper’s method is to compare the simulation against a simple statistical baseline. We had never run one. This arm answers the same cases from the grounding layer alone — the best-matching published national figure at the cutoff, applied identically to every segment — with no persona reasoning and no LLM call. Because the same national figure is used for every segment, the baseline cannot distinguish segments at all; the council has to beat it on segment-level error and on ranking the right segment.
Stats-only baseline MAE
11.0 pts
Council MAE
5.9 pts
Segment readings scored
21
The baseline uses only percentage-scale published statistics, so it is on the same scale as the observed segment figures. The two Irish trust questions are 0–10-scale items with no published percentage anchor and are excluded from this arm, not estimated.
Cases are Irish and UK decisions with outcomes already on the public record. The list cannot be trimmed after results are known.
| Case | Market | Evidence cutoff | Predicted | Observed | Source |
|---|---|---|---|---|---|
| Share of the Irish electorate voting Yes in the 2024 Family amendment referendum | IE | 2024-03-07 | — | — | Referendum Returning Officer, Ireland |
| Share of the Irish electorate voting Yes in the 2024 Care amendment referendum | IE | 2024-03-07 | — | — | Referendum Returning Officer, Ireland |
| Share of in-scope containers returned in the first full year of Re-turn | IE | 2024-02-01 | — | — | Re-turn annual reporting |
| Share of vehicles compliant on the day ULEZ expanded to outer London | UK | 2023-08-28 | — | — | Transport for London compliance reporting |
| Direction and magnitude of the CSO consumer-sentiment series over the next quarter | IE | 2025-06-30 | — | — | CSO consumer-sentiment series |
Indicative market ranges for comparison only, not quotations. Outweigh’s own pricing is published in full on the pricing page.
The fastest way to test this is on one of your own calls: freeze the prediction, act, and review the record together at 90 days.