Forecaster Arena – Testing LLMs on real events with prediction markets

Show HN: Forecaster Arena – Testing LLMs on real events with prediction markets

Forecaster Arena – Testing LLMs on real events with prediction markets operates in show hn launch.

Profile

Hey HN! I'm Mert.I built this because I was frustrated with LLM benchmarks potentially being contaminated by training data. When a model scores 99.9% on MMLU-Pro-Max, we can't tell if that's genuine reasoning or memorization.Forecaster Arena tries to solve this by testing models on events that haven't happened yet—real prediction markets from Polymarket. The ground truth is reality itself, weeks or months later.How it works:7 frontier LLMs (GPT-5.1, Claude Opus 4.5, Gemini, Grok, DeepSeek, etc.) (will be updated) -> Each gets $10k virtual capital weekly -> They bet on 500+ real prediction markets -> Bet size = confidence (larger bet = more confident) -> We measure calibration (Brier score) +

As a business in show hn launch, Forecaster Arena – Testing LLMs on real events with prediction markets is listed here with the public details available about it.

Directory data places it at the Show HN launch stage.

Website: forecasterarena.com