Frequently asked questions

What is SWE-bench Verified and why does it matter?

SWE-bench Verified is a benchmark of 500 human-verified real GitHub issues. Models receive a repository and a failing test — they must produce a code patch that makes the test pass. It is the closest benchmark to real-world software engineering work. As of mid-2026, Claude Fable 5 leads at 95%, followed by GPT-5.5 (88.7%) and Claude Opus 4.8 (88.6%).

What do the SR and 3P labels mean on the leaderboard?

SR means the score was self-reported by the model's provider in their official model card, technical report, or launch blog. 3P means the score was independently measured by a third party such as Artificial Analysis, the ARC Prize Foundation, or NIST/CAISI. Third-party scores are more reliable for head-to-head comparisons because they use standardised evaluation conditions.

Why does DeepSeek V4-Pro show two different SWE-bench scores?

DeepSeek V4-Pro self-reported 80.6% on SWE-bench Verified, but NIST/CAISI independently measured 74.0% — a 6.6-point gap. The difference is attributed to scaffolding setup, system-prompt construction, and token-budget settings. We show both numbers on the leaderboard so you can evaluate the discrepancy yourself.

What is GPQA Diamond?

GPQA Diamond is a benchmark of graduate-level questions in biology, chemistry, and physics, written by domain experts so that Google search cannot answer them. As of June 2026, Claude Opus 4.8 and GPT-5.5 are tied at 93.6%, followed by Qwen 3.7-Max (92.4%) and Claude Opus 4.6 (91.3%). Human PhD experts score ~65% on the same questions.

Which AI model is best for coding tasks in FlowHunt?

For real-world GitHub-style coding (SWE-bench Verified), Claude Fable 5 leads at 95% and GPT-5.5 at 88.7%. For competitive programming (LiveCodeBench), DeepSeek V4-Pro leads at 93.5%. For agentic multi-language coding (SWE-bench Pro), Qwen 3.7-Max leads at 60.6%. All models marked 'in FlowHunt' are directly available in the platform.

What is ARC-AGI-2 and who leads it?

ARC-AGI-2 is a benchmark of novel visual pattern recognition tasks that cannot be solved by memorising training data, independently verified by the ARC Prize Foundation. As of June 2026, Claude Opus 4.6 leads at 69.17% (3P verified), followed by Claude Sonnet 4.6 (58.3%). Human baseline is ~60%.

What is the Artificial Analysis (AA) Intelligence Index?

The Artificial Analysis Intelligence Index is an independent composite score not self-reported by any provider. Current leaders: Claude Opus 4.8 (61.4 pts), GPT-5.5 (60.2 pts), Claude Opus 4.7 (57.3 pts), Qwen 3.7-Max (56.6 pts), DeepSeek V4-Pro (52 pts). These are directly comparable since they are measured under identical conditions.

What is the difference between open-weight and closed AI models?

Open-weight models publish their trained weights for self-hosting. Closed models are API-only. Open-weight frontier models on this leaderboard include DeepSeek V4-Pro (49B/1.6T MoE), MiniMax M3, Nemotron 3 Ultra (55B/550B MoE), Mistral Large 3 (41B/675B MoE), and all Meta Llama 4 variants. Open-weight models offer data privacy, customisation, and no per-token costs.

How often is this leaderboard updated?

The leaderboard is updated when significant new benchmark results are published or independently verified. Scores are current as of June 12, 2026. Every score is tagged SR (self-reported) or 3P (third-party) so you always know the source. Each model also has its own page with a deeper benchmark breakdown.