AskNews
Emergent Methods

How TU Delft tested news-grounded LLMs against prediction markets

How TU Delft tested news-grounded LLMs against prediction markets — AskNews research article card

What if you could feed a language model the latest news about an upcoming election, a product launch, or an interest-rate decision, and ask it to estimate the probability of each outcome? Would those estimates be any good?

A language model can summarize news, but converting a summary into a calibrated probability is a different problem. Prediction markets combine the judgments of many participants who back their beliefs with real money, and the resulting prices update continuously as new information arrives. Whether a model given relevant news can match that collective, incentive-driven signal is an empirical question, and one worth testing carefully.

A TU Delft master's thesis, Performance of Large Language Models in Prediction Markets, set out to do exactly that. Ketill Hugi Halldórsson built a pipeline that retrieved news relevant to each of 53 live Polymarket markets through the AskNews API, summarized it into a compact factual briefing with explicit instructions not to predict, and handed the same summary to every model. Six models, selected in 2025 and not the newest available even then, were each asked to produce a single probability for each market on collection days under both zero-shot and chain-of-thought prompting. The pipeline code is publicly available.

What the results show

Forecasting accuracy is measured by the Brier score, where a lower value means the forecast was closer to what actually happened. Across all six models and both prompting strategies, Polymarket's implied probabilities consistently scored lower than any individual model's forecasts. The market was simply more accurate.

A natural next question is whether combining several models helps. If different models make different errors, averaging their forecasts might cancel some of those errors out. Halldórsson tested an equal-weight ensemble of all six models, but the averaged forecast still fell short of the market.

What if the combination included the market price itself? A regression-based hybrid the thesis calls the Iron Man model learned weights for each model's forecast alongside the market probability. In sample, it achieved near parity with the market. But fitting training data well is not the same as predicting new events. To test generalization, the thesis used a leave-one-market-out design: for each of the 53 markets, the model was trained on all remaining markets and then asked to forecast the held-out one. This is not a chronological forward split; it removes the target market entirely from training.

Forecasting accuracy was only one test. The thesis also asked whether the forecasts could support profitable trades, and the contrast in simulated profit was stark. Across 159 one-share frictionless simulated trades (53 markets evaluated at three points each, held to resolution), the in-sample hybrid earned a total of $15.58. The held-out version earned $1.10. A simple strategy that just bought the market favourite, with no model or news input at all, earned $4.38 over the same trades. These are dollar totals from a simplified simulation, not investment returns.

The evidence supports treating news-grounded LLM forecasting as a research tool worth studying, not as a replacement for prediction markets.

The thesis is careful about what it can and cannot say. These were six selected models tested across a single sample of markets and a limited time window. There was no experimental arm that removed the news retrieval layer, so the study evaluates the complete grounded pipeline rather than isolating what the news contributed on its own. Exact numerical replication depends on API access and matching data collection conditions.

Explore forecasting data through the AskNews news documentation.