Leaderboard
Main capability ranking. Our hardest, sealed questions.
Editions over time
Edition v5 · cut 7/28/2026 · 40 models · 109 questions · 63 cells imputed (worst-of-others)
Per-model aggregate
Mean Cramér-log is the ranking metric (lower is better), scored against the verified ground-truth distribution. The Answered column shows real vs imputed; a malformed or unscorable answer counts as unanswered.
| # | Model | Mean Cramér | Answered | Ran |
|---|---|---|---|---|
| 1 | anthropic/claude-opus-5 | 0.2939 | 109/109 | 7/28/2026 |
| 2 | meta/muse-spark-1.1 | 0.3090 | 109/109 | 7/28/2026 |
| 3 | openai/gpt-5.6-sol | 0.3157 | 109/109 | 7/28/2026 |
| 4 | openai/gpt-5.5 | 0.3245 | 109/109 | 7/28/2026 |
| 5 | anthropic/claude-opus-4.8 | 0.3309 | 109/109 | 7/28/2026 |
| 6 | openai/gpt-5.6-terra | 0.3625 | 109/109 | 7/28/2026 |
| 7 | openai/gpt-5.6-terra-pro | 0.3628 | 109/109 | 7/28/2026 |
| 8 | google/gemini-3.5-flash | 0.3770 | 109/109 | 7/29/2026 |
| 9 | openai/gpt-5.6-luna | 0.3785 | 109/109 | 7/28/2026 |
| 10 | openai/gpt-5 | 0.3899 | 109/109 | 7/28/2026 |
| 11 | qwen/qwen3.7-max | 0.3975 | 109/109 | 7/28/2026 |
| 12 | z-ai/glm-5.2 | 0.3993 | 109/109 | 7/28/2026 |
| 13 | openai/gpt-5.1 | 0.4011 | 109/109 | 7/28/2026 |
| 14 | deepseek/deepseek-v4-pro | 0.4029 | 109/109 | 7/28/2026 |
| 15 | openai/o3 | 0.4033 | 109/109 | 7/28/2026 |
| 16 | moonshotai/kimi-k3 | 0.4117 | 108/109 · 1 imp | 7/28/2026 |
| 17 | x-ai/grok-4.5 | 0.4193 | 109/109 | 7/28/2026 |
| 18 | moonshotai/kimi-k2.6 | 0.4369 | 108/109 · 1 imp | 7/28/2026 |
| 19 | moonshotai/kimi-k2-thinking | 0.4433 | 109/109 | 7/28/2026 |
| 20 | ~google/gemini-pro-latest | 0.4546 | 109/109 | 7/28/2026 |
| 21 | anthropic/claude-sonnet-5 | 0.4615 | 109/109 | 7/28/2026 |
| 22 | google/gemini-3.1-pro-preview | 0.4641 | 109/109 | 7/29/2026 |
| 23 | minimax/minimax-m2.7 | 0.4722 | 109/109 | 7/28/2026 |
| 24 | moonshotai/kimi-k2 | 0.4895 | 109/109 | 7/28/2026 |
| 25 | qwen/qwen3-235b-a22b-thinking-2507 | 0.5475 | 109/109 | 7/28/2026 |
| 26 | mistralai/mistral-medium-3-5 | 0.5590 | 109/109 | 7/29/2026 |
| 27 | openai/gpt-5-mini | 0.5752 | 109/109 | 7/28/2026 |
| 28 | x-ai/grok-4.3 | 0.5920 | 109/109 | 7/29/2026 |
| 29 | deepseek/deepseek-r1-0528 | 0.6227 | 109/109 | 7/28/2026 |
| 30 | deepseek/deepseek-r1 | 0.6284 | 109/109 | 7/28/2026 |
| 31 | mistralai/mistral-large-2512 | 0.7004 | 109/109 | 7/28/2026 |
| 32 | openai/gpt-oss-120b | 0.7169 | 109/109 | 7/29/2026 |
| 33 | deepseek/deepseek-v3.2 | 0.7722 | 103/109 · 6 imp | 7/28/2026 |
| 34 | minimax/minimax-m3 | 0.7855 | 109/109 | 7/28/2026 |
| 35 | anthropic/claude-haiku-4.5 | 0.8382 | 109/109 | 7/28/2026 |
| 36 | minimax/minimax-m2 | 0.9556 | 108/109 · 1 imp | 7/28/2026 |
| 37 | meta-llama/llama-4-maverick | 0.9833 | 108/109 · 1 imp | 7/28/2026 |
| 38 | openai/gpt-oss-20b | 1.1611 | 109/109 | 7/28/2026 |
| 39 | anthropic/claude-fable-5 | 1.9182 | 63/109 · 46 imp | 7/28/2026 |
| 40 | meta-llama/llama-4-scout | 2.2418 | 102/109 · 7 imp | 7/28/2026 |
Scores over time
Mean Cramér vs. release date. No per-question breakdown here — the held-out set stays sealed, so each dot’s hover card shows only its aggregate score.
- OpenAI
- Anthropic
- Google DeepMind
- DeepSeek
- Meta
- Other
| Model | Lab | Release date | Mean Cramér-log | Answered |
|---|---|---|---|---|
| DeepSeek R1 | DeepSeek | Jan 20, 2025 | 0.628 | 109/109 |
| Llama 4 Scout | Meta | Apr 5, 2025 | 2.242 | 102/109 |
| Llama 4 Maverick | Meta | Apr 5, 2025 | 0.983 | 108/109 |
| o3 | OpenAI | Apr 16, 2025 | 0.403 | 109/109 |
| Deepseek r1 0528 | DeepSeek | May 28, 2025 | 0.623 | 109/109 |
| Kimi k2 | — | Jul 11, 2025 | 0.490 | 109/109 |
| qwen3 235b a22b Thinking 2507 | — | Jul 25, 2025 | 0.548 | 109/109 |
| Gpt Oss 20b | OpenAI | Aug 5, 2025 | 1.161 | 109/109 |
| Gpt Oss 120b | OpenAI | Aug 5, 2025 | 0.717 | 109/109 |
| Gpt 5 Mini | OpenAI | Aug 7, 2025 | 0.575 | 109/109 |
| Gpt 5 | OpenAI | Aug 7, 2025 | 0.390 | 109/109 |
| Claude Haiku 4.5 | Anthropic | Oct 15, 2025 | 0.838 | 109/109 |
| Minimax m2 | — | Oct 23, 2025 | 0.956 | 108/109 |
| Kimi k2 Thinking | — | Nov 6, 2025 | 0.443 | 109/109 |
| GPT-5.1 | OpenAI | Nov 13, 2025 | 0.401 | 109/109 |
| Deepseek v3.2 | DeepSeek | Dec 1, 2025 | 0.772 | 103/109 |
| Mistral Large 2512 | — | Dec 1, 2025 | 0.700 | 109/109 |
| Gemini 3.1 Pro Preview | Google DeepMind | Feb 19, 2026 | 0.464 | 109/109 |
| Minimax m2.7 | — | Mar 18, 2026 | 0.472 | 109/109 |
| Kimi k2.6 | — | Apr 20, 2026 | 0.437 | 108/109 |
| Deepseek v4 Pro | DeepSeek | Apr 24, 2026 | 0.403 | 109/109 |
| Gpt 5.5 | OpenAI | Apr 24, 2026 | 0.325 | 109/109 |
| Gemini Pro Latest | — | Apr 27, 2026 | 0.455 | 109/109 |
| Mistral Medium 3 5 | — | Apr 30, 2026 | 0.559 | 109/109 |
| Grok 4.3 | — | Apr 30, 2026 | 0.592 | 109/109 |
| Gemini 3.5 Flash | Google DeepMind | May 19, 2026 | 0.377 | 109/109 |
| qwen3.7 Max | — | May 21, 2026 | 0.398 | 109/109 |
| Claude Opus 4.8 | Anthropic | May 27, 2026 | 0.331 | 109/109 |
| Minimax m3 | — | May 31, 2026 | 0.786 | 109/109 |
| Claude Fable 5 | Anthropic | Jun 9, 2026 | 1.918 | 63/109 |
| Glm 5.2 | — | Jun 16, 2026 | 0.399 | 109/109 |
| Claude Sonnet 5 | Anthropic | Jun 30, 2026 | 0.462 | 109/109 |
| Grok 4.5 | — | Jul 8, 2026 | 0.419 | 109/109 |
| Gpt 5.6 Sol | OpenAI | Jul 9, 2026 | 0.316 | 109/109 |
| Gpt 5.6 Terra | OpenAI | Jul 9, 2026 | 0.363 | 109/109 |
| Gpt 5.6 Terra Pro | OpenAI | Jul 9, 2026 | 0.363 | 109/109 |
| Gpt 5.6 Luna | OpenAI | Jul 9, 2026 | 0.379 | 109/109 |
| Muse Spark 1.1 | — | Jul 16, 2026 | 0.309 | 109/109 |
| Kimi k3 | — | Jul 16, 2026 | 0.412 | 108/109 |
| Claude Opus 5 | Anthropic | Jul 24, 2026 | 0.294 | 109/109 |