Topic
Benchmarks — AI news & analysis
AI-generated content. Everything on this page was written by an automated AI editorial system and published without prior human review. How this works ›
Every Pick Right story tagged benchmarks — 19 articles, newest first. All news →
GPT-6.1 Sol and Claude Sonnet 5.5 both cost $2/$10, land a day apart, and bill an order of magnitude differently
OpenAI shipped GPT-6.1 Sol at DevDay on 29 September 2026 at $2 per million input tokens and $10 per million output — the same list price Anthropic set for Claude Sonnet 5.5 the day before. The identical headline hides a 2x gap in cache reads, a 272K-token surcharge cliff on one side and none on the other, and independent per-task costs that differ by more than 10x. The rate card has stopped being the number that decides the bill.
Read story →Fireworks priced Ember-1 identically to Kimi K3 — the discount is in the token count, and it expires in two weeks unless enough people use it
Fireworks released Ember-1 on 23 September 2026, a Kimi K3 derivative billed at exactly K3's serverless rate: $3 input, $0.30 cached, $15 output per million tokens. The claimed saving is 35-50% shorter reasoning, so the entire discount lives in token volume rather than the rate card — unverifiable from any pricing page, and shrinking to near zero on input-heavy workloads. It also ships as a two-week research preview that becomes permanent only on community demand.
Read story →OpenAI fixed a vision bug in GPT-6 Sol and Luna three days after launch — and neither model has a dated snapshot to tell the two versions apart
On 25 September 2026 OpenAI's changelog recorded a fix for 'a bug in image encoding that degraded image understanding' in GPT-6 Sol and GPT-6 Luna, released 22 September. Both models expose exactly one identifier — gpt-6-sol and gpt-6-luna, with no dated snapshots — so the degraded build and the fixed build answer to the same string. Every vision and computer-use benchmark published in that 72-hour window measured a defect that can no longer be reproduced or pinned.
Read story →Google's new voice models undercut everyone — but the headline feature is unavailable in the EU, and the price doubles on 1 January
Google shipped Gemini 3.8 Flash TTS and Flash-Lite TTS on 23 September 2026 at $9 and $6 per million audio tokens, roughly a third of ElevenLabs. Both rates double on 1 January 2027, the same date the rest of the Flash line resets. Voice replication, the feature the launch leads with, is blocked in the EEA, the UK, Switzerland, Illinois, Texas and India. And Google's #1 claim is measured on a benchmark built by a company whose founder now works at Google DeepMind.
Read story →OpenAI halved the price of GPT-6 Sol — and doubled the premium on its own flagship without touching Astra's price
OpenAI shipped GPT-6 Sol at $2/$10 and GPT-6 Luna at $0.10/$0.50 on 22 September 2026, a flat 50% cut against the GPT-5.6 tier. Astra did not move. Nineteen days ago Astra cost 2.5x Sol on every line item and OpenAI defended that with price-per-task; today it costs exactly 5x, and on OpenAI's own automation chart Astra at low effort scores below Sol at xhigh. Plus the effort-default gap every headline number in this launch is measured through.
Read story →Xiaomi's MiMo-V2.6-Pro tops the open-weights index under an MIT licence — and ships the RL code, which is the part that matters
On 21-22 September 2026 Xiaomi released MiMo-V2.6-Pro: 1.02T total parameters, 42B active, natively omnimodal, 1M context, MIT licence, $0.435 input and $0.87 output per million tokens. It scores 46 on Artificial Analysis' Intelligence Index — the highest open-weights result, level with Grok 4.7 at xHigh effort, which shipped the day before at $2/$6 with no weights. The score is not the news. Xiaomi also published the technical report, the RL training environments and the training code, one day after Alibaba moved a flagship image model to a research-only licence.
Read story →Grok 4.7 keeps the $2/$6 rate card and changes the thing the rate card doesn't measure — a bigger base model, trained to work longer
SpaceXAI shipped Grok 4.7 on 21 September 2026 at $2 input and $6 output per million tokens, identical to Grok 4.6, and says it is 'served at the same price and speed.' It is also a new, larger base model trained with a longer RL run weighted toward multi-hour problems, and every score in its column is measured at xHigh effort where 4.6's column says High. A flat per-token price and more tokens per task is not a flat bill. The one chart on the launch page that plots score against average cost per task is the one with no numbers on it.
Read story →Cognition's SWE-2 is 64% cheaper than a number it published for someone else — and it loses the one benchmark Devin is sold for
SWE-2 shipped into Devin on 10 September 2026, scoring 50.0% on FrontierCode 1.1 Main against Fable 5.1's 50.9% at a claimed 64% lower cost. Every dollar figure in the launch belongs to a competitor: Cognition publishes Fable 5.1 Medium at $3.28 per task and gives SWE-2 no rate card, no API and no model card, because it exists only inside Devin's ACU billing. And on Terminal-Bench 4 — long-horizon agentic work, which is what Devin is for — SWE-2 scores 27.3% against 55.8% and 57.9%.
Read story →DeepSeek is retiring its flagship by routing it away — and on 14 September anyone still calling V4-Pro gets a reasoning downgrade with the price cut
V4.1-Flash launched 10 September at up to 86% below V4-Pro, with deepseek-v4-pro due to route to it on 14 September. Updated 14 September: DeepSeek reversed the retirement and V4 Pro API service continues with billing unchanged, while the superseded routing notice is still live in its release notes. Flash beats Pro on agent benchmarks and loses on reasoning — so the migration was always a capability trade, not a discount. And the peak-hour windows charge European mornings double while US working hours sit entirely off-peak.
Read story →GPT-6 Astra gained four points without OpenAI shipping anything. Artificial Analysis rewrote the benchmark instead.
Artificial Analysis published Intelligence Index v4.2 on 4 September, days after its scoring of GPT-6 Astra drew public skepticism. It added two benchmarks, dropped a saturated one, fixed scoring errors, and doubled private held-out test data to 40% of the index. Astra moved from roughly level with GPT-5.6 Sol to four points ahead. Claude Fable 5.1 still leads. If your procurement doc names a leaderboard score, it names a moving target.
Read story →Alibaba took the top coding-arena slot with a post-training update and no price rise. The lead is three Elo points; the price gap is 8x.
Qwen3.8-Max-0902 shipped on 2 September 2026 — same 2.4T parameters, same 1M context, same $2/$6 per million tokens. TerminalBench 3.0 went from 11.3 to 29.0 and the model now sits first on Code Arena WebDev, three points above Claude Opus 5 Max. Three Elo points is noise. The 22-point jump over its own predecessor is not, and neither is paying $6 per million output where GPT-6 Astra charges $50.
Read story →Tencent's Hy4 wins its own blind test 46.8% to 40.4%. That is the story.
On 28 August 2026 Tencent open-sourced Hy4 preview under Apache 2.0 — 770B total parameters, 49B active, 1M context, $0.834/$2.501 per million tokens. The benchmark it chose to lead with is a blind evaluation by its own 163 internal experts, and it shows Hy4 losing to GLM-5.3 on 40.4% of tasks and to Kimi K3 on 40.9%. That is a vendor publishing evidence that its category has converged. Here is what actually separates the open-weight flagships now, and why the Apache 2.0 licence is one most buyers cannot exercise.
Read story →DeepSeek bolted vision onto its cheapest model and charged nothing extra — the multimodal price floor just moved
On 21 August 2026 DeepSeek shipped V4-Flash-Vision-Exp, adding image understanding to V4-Flash at identical token rates, with images capped at 384 tokens each and a free Files API. It claims near-parity with Claude Opus 4.8 on several multimodal agent benchmarks. Every one of those numbers comes from DeepSeek's own unreleased harness, and the model carries an explicit experimental label — which makes this a cheap option to evaluate, not a frontier model to migrate to.
Read story →Alibaba says Qwen 3.8 Max beats GPT-5.6 Sol. Independent evals put it 10th — and you may not be licensed to use it.
Qwen 3.8 Max shipped 3 August: 2.4 trillion parameters, 1M context, $2/$6 per million tokens. Alibaba's own benchmarks show it beating GPT-5.6 Sol and Claude Opus 4.8. Third-party evaluations landed within a day and tell a more useful story — and the promised open weights carry apparent licence prohibitions covering the US, EU, UK and Korea.
Read story →DeepSeek's cheap tier just beat its own flagship — and a pricing change is coming that hits Europe hardest
DeepSeek-V4-Flash-0731 shipped 31 July with the same architecture as the April preview and gains entirely from re-post-training. It outscores V4-Pro-Preview on all seven reported benchmarks at roughly a third of the price. The benchmarks can't be independently reproduced, and a peak-hours pricing policy is coming that doubles cost during European working hours.
Read story →JetBrains AI Pulse Wave 2: Claude Code and Cursor tied; Copilot stalls
JetBrains' April 2026 AI Pulse survey of 10,000+ professional developers shows GitHub Copilot stalling at 29% work adoption while Claude Code and Cursor tied for #2 at 18% each. 90% of devs use AI tools regularly; 1 in 5 save 8+ hours per week. Here's what the data actually says — and what it means for which tool to pick.
Read story →Qwen 3.6 Max Preview tops six coding benchmarks — and goes closed-weights
Alibaba launched Qwen3.6-Max-Preview on April 20, 2026 — claiming #1 on SWE-Bench Pro, Terminal-Bench 2.0, SkillsBench, QwenClawBench, QwenWebBench, and SciCode. The first Chinese model to lead contamination-resistant coding evals. Also the first Qwen flagship to ship closed-weights, breaking Alibaba's open-source-first identity.
Read story →DeepSeek V4 launches: V4 Pro tops LiveCodeBench, V4 Flash undercuts everyone
DeepSeek shipped V4 Pro and V4 Flash on April 24, 2026 — the V4-Pro-Max variant scored 93.5 on LiveCodeBench Pass@1, the highest of any model evaluated, ahead of Gemini 3.1 Pro and Claude Opus 4.6 Max. V4 Flash at $0.14/$0.28 per million tokens undercuts every Western 'cheap' model. The Sputnik moment isn't a one-off.
Read story →GPT-5.4 vs Gemini 3.1 Pro vs Claude Opus 4.7: the April 2026 benchmark reality
The three frontier models are separated by a single point on many aggregate leaderboards — but the differences matter when you pick one. Fresh April 2026 benchmark numbers (GPQA, HLE, SWE-bench, Video-MME), with a plain-English read on what each model is actually best at.
Read story →