Oct 29, 2026 · 27 min
Benchmark overhaul widens AI’s open-source divide
The AI Rankings Just Got Reshuffled
The episode examines whether a revised benchmark is clarifying model capabilities or merely changing the story rankings tell about usefulness.
- 1A major benchmark revision reshuffles model rankings and makes the gap between open- and closed-source systems appear wider.
- 2Benchmark providers increasingly influence how AI companies and audiences define progress, not merely how they measure it.
- 3The episode places benchmark uncertainty alongside Google and Microsoft cloud strength and growing antitrust scrutiny of Nvidia.
Don't miss
The benchmark overhaul reshuffles model rankings and makes open-source systems appear further behind closed-source competitors.
The brief
Nathaniel Whitmore opens with a basic problem in AI discourse: benchmark scores have become shorthand for capability, even as benchmark providers gain more power over the comparison itself.
A major overhaul changes how models are tested and reshuffles the rankings, particularly widening the apparent gap between open- and closed-source systems.
The central tension is whether the revised leaderboard measures meaningful progress or simply reflects the assumptions, incentives, and limits built into the test.
Humanity's Last Exam appears in the discussion as an example of ambitious evaluation designed to push beyond conventional measures of model performance.
The benchmark debate sits within a broader AI market story: Google and Microsoft report strong cloud revenue while Nvidia faces expanding antitrust scrutiny.
What was said on this episode
37 statements · 15 positive · 10 negative · 5 mixed · 7 neutral
AI benchmarks become more important as AI models gain significance
“as AI models become more significant, the benchmarks that tell us how they perform relative to one another consequently get more important”
Listen at 0:19
Gemini 4 Carbon significantly improves on Gemini 4 Argon in benchmarks
“when it comes to the benchmarks, the answer is a definitive yes”
Listen at 2:28
Gemini 4 Carbon leads six of eight key benchmarks.
“On six of those eight key benchmarks, Gemini 4 Carbon is now the top of the leaderboard, ahead of Astra and Opus 5.5.”
Listen at 2:46
Gemini 4 Carbon leads Artificial Analysis Intelligence Index V5
“Gemini 4 Carbon is now the frontrunner on the new Artificial Analysis Index V5”
Listen at 2:54
Gemini 4 Carbon could return Google to frontier-model competition
“This release, if people's experience with it matches the benchmarks, puts Google genuinely back in the race for the Frontier”
Listen at 3:33
Gemini 4 Carbon’s cost is roughly unchanged from Gemini 4 Argon.
“Cost is basically unchanged compared to Argon”
Listen at 3:40
Gemini 4 Carbon’s task cost is below Opus 5.5 and above Astra
“that's 23% cheaper than Opus 5.5 and 20% more expensive than Astra”
Listen at 3:47
Gemini 4 Carbon trails rivals for 3D design use cases.
“at least when it comes to that particular use case, Carbon is not at the same level as its rivals.”
Listen at 4:46
Gemini 4 Carbon trails rivals for 3D design use cases
“Carbon is not at the same level as its rivals”
Listen at 4:47
Gemini 4 Carbon appears promising pending hands-on evaluation
“it looks promising and that in and of itself is exciting”
Listen at 6:32
Google may need to cannibalize its search business through AI.
“Google might have reached the moment where they're forced to cannibalize their own search business”
Listen at 7:56
AI infrastructure buildout may have reached peak growth speed.
“So it looks like the AI buildout may have reached peak velocity.”
Listen at 8:59
AI infrastructure expansion may have reached peak velocity
“it looks like the AI buildout may have reached peak velocity”
Listen at 8:59
Hyperscalers want faster data-center expansion but face physical constraints
“the hyperscalers, if they had their druthers, would like to continue accelerating their data center build out, but are running into more fundamental and physical constraints”
Listen at 9:32
The DOJ appears to be conducting a comprehensive Nvidia antitrust investigation.
“it now seems the DOJ is embarking on a full top-to-bottom antitrust investigation of NVIDIA.”
Listen at 11:22
DOJ’s Nvidia investigation covers the company broadly
“the DOJ is embarking on a full top-to-bottom antitrust investigation of NVIDIA”
Listen at 11:23
The DOJ investigation could end without enforcement action
“the investigation could conclude with no action”
Listen at 11:30
AI benchmarks are limited but important
“I think that benchmarks are inherently limited, but also important”
Listen at 15:25
Without benchmarks, model comparisons would rely mainly on subjective impressions.
“Without them, we don't have good tools other than Strictly vibes to understand the difference between different models”
Listen at 15:29
AI benchmark results should not be treated as definitive evidence of model quality.
“in almost no circumstances do I think we should take the performance of one model on any one benchmark or even in the aggregate as the gospel.”
Listen at 15:35
Well-designed benchmarks help identify where models fit in an AI stack.
“Done well, they are a useful guide to helping us understand where different models fit into our overall stack.”
Listen at 15:42
Well-designed benchmarks help position AI models within technology stacks
“they are a useful guide to helping us understand where different models fit into our overall stack”
Listen at 15:43
Artificial Analysis changes benchmarks and weightings as model abilities evolve
“artificial analysis also evolved, switching out benchmarks and rebalancing their weightings as different model abilities became comparatively more important”
Listen at 17:12
Benchmark maxing can inflate model rankings beyond real-world performance
“benchmark maxing allowed developers to game the system and rank much higher than real-world testing would support”
Listen at 17:30
Private benchmark datasets make it harder for developers to optimize specifically for tests.
“The idea is to make benchmark maxing more difficult, as model developers theoretically can't train their models to the test.”
Listen at 18:32
Agentic work is a major focus of the redesigned Intelligence Index.
“Agentic work is the other huge change that's evident from the redesign of the index.”
Listen at 20:09
Artificial Analysis V5 coding benchmarks measure only agentic performance.
“The coding benchmarks included in the index now only measure agentic performance”
Listen at 20:13
Artificial Analysis V5 equally weights four capability categories
“artificial analysis has chosen to equally weight the index across four categories”
Listen at 20:26
Opus 5.5 led Artificial Analysis V5 with a score of 58.
“In version 5, Opus 5.5 was still in the lead with a score of 58”
Listen at 20:47
Opus 5.5 led Artificial Analysis V5 with 58 points before Carbon
“Opus 5.5 was still in the lead with a score of 58”
Listen at 20:49
GPT-6 Astra ranked second on Artificial Analysis V5 with 57 points
“GPT-6 Astra jumped into then 2nd place with a score of 57”
Listen at 20:54
V5 substantially widened the gap between open-weight and US proprietary models.
“the gap between open-weight models and US proprietary models had opened up massively after the index rebalancing.”
Listen at 21:45
GPT-61 Terra is cheapest and Fable 5-1 most expensive among top ten models
“GPT-61 Terra as the cheapest at $0.96 per task, and Fable 5-1 as the most expensive at $13.04 per task”
Listen at 22:52
Benchmark category weighting will become a major debate.
“I think one of the biggest debates will become what the relevant weighting should actually be.”
Listen at 25:03
AI rankings should be organized by model and effort level
“people might be better served by the ability to organize the index, not just by model, but also by effort”
Listen at 25:51
Artificial Analysis Progress Index tracks leading-model capability improvements since November 2022.
“The index places all models released since November 2022 on a capability scale to see how each lab's leading model improves over time.”
Listen at 26:03
AI benchmarks are important but inherently limited
“benchmarks are important and inherently limited”
Listen at 26:38
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Weekly email
State of the Pod, every Monday
The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.