Oct 7, 2026 · 50 min
Listen from 36:44
Listen at 36:44
Test AI Models Against Real Work, Not Vendor Benchmarks
The Best Way to Test New AI Models
As models multiply and switching carries hidden costs, teams need evaluations that reflect actual tasks, constraints, and changing expectations.
- 1Repeatable evaluations start with realistic tasks, measurable prompts, explicit constraints, and clear rejection criteria.
- 2Model choices should account for quality, speed, effort settings, security, cost, and the practical burden of switching.
- 3Teams can switch models, split usage across tools, or stay put as capabilities and user expectations evolve.
Don't miss
Nufar explains why the final evaluation should produce a practical choice—switch, split usage, or stay—rather than a simplistic leaderboard ranking.
The brief
With frontier, efficient, and open-weight models arriving quickly, vendor benchmarks offer an incomplete guide to choosing what belongs in real workflows.
Nufar Gaspar proposes a repeatable five-step evaluation process built around standardized tasks, measurable prompts, consistent settings, and evidence from actual work.
The decision is not simply which model scores highest: teams must weigh quality against speed, effort, security, cost, and the friction of switching.
A particularly useful distinction is between switching entirely, splitting usage across tools, and staying with the current setup when gains do not justify disruption.
The framework also has to evolve, because exposure to faster systems changes expectations and makes speed itself part of perceived model quality.
What was said on this episode
34 statements · 22 positive · 5 negative · 1 mixed · 6 neutral
Benchmark outputs should be anonymized to reduce evaluator bias.
“we'll be advised to hide the names of which models or which tools created each result because we are very biased towards our beloved tools.”
Listen at 6:15
A benchmark win does not automatically justify changing the current AI setup.
“the fact that in a specific benchmark run, one tool or one model outperformed your current setup does not necessarily mean that automatically you need to make changes.”
Listen at 6:31
New AI models should be judged after real-world use over time.
“Eventually, we need to see what happens after a while.”
Listen at 8:23
Current frontier AI models are highly capable.
“the frontier is very good”
Listen at 9:29
AI model benchmarking need not be conducted weekly because frequent testing creates confusion.
“it doesn't have to be on a weekly basis because that in itself can really create a lot of confusion.”
Listen at 10:08
Frequently performed tasks offer the greatest potential benefit from model improvements.
“those are where you get the highest uplift.”
Listen at 12:21
Personal benchmarks should include one or two previously unsuccessful wishlist tasks.
“I would strongly recommend that you have one or two of those”
Listen at 13:16
A personal AI benchmark should contain roughly four to six use cases.
“My recommendation would be like four to six type of such use cases”
Listen at 13:34
Benchmarking should compare the existing model with up to four candidates.
“I would recommend to run the existing model versus a new one or up to four different candidates.”
Listen at 15:11
Each benchmark request should run in a fresh chat to avoid context effects.
“we want to run every request in a fresh chat such that the context window will not send us on a tailspin”
Listen at 15:23
Relevant personal benchmarks should include visual tasks.
“I strongly recommend that some of the tasks will be more visual”
Listen at 17:45
An AI evaluation judge should itself be evaluated for suitability.
“you should evaluate the judge itself.”
Listen at 20:02
The judging model should ideally differ from the models being evaluated.
“the model that you will use for judging is different from the models that executed the results.”
Listen at 20:26
High-stakes benchmarks should repeat each request several times per model.
“if you're running this personal benchmark on something that is very critical and there is perhaps a monetary implication or other implications to you making changes to how you work, ideally you should run the same request several times per model”
Listen at 21:17
Critical benchmarks should run each request multiple times per model.
“ideally you should run the same request several times per model”
Listen at 21:26
Identical model configurations and prompts can produce different outputs.
“Even if we're using the exact same model with the exact same configuration and the exact same prompt, you will get different results.”
Listen at 21:36
Without a significant performance gain, retaining existing AI habits can be preferable.
“Habit formation is an expensive thing, so if there is no significant boost, just staying with the way you used to work also has merits.”
Listen at 26:17
Users should change AI tools when benchmark results and practical considerations justify it.
“if all of these tests justify the change, then make the change”
Listen at 26:25
Sensitive company data should not be sent through OpenRouter.
“if your company is sensitive, don't use something like Open Router with company data.”
Listen at 31:27
Fable’s total benchmark cost was substantially higher than the alternatives.
“the total cost of Fable is so much higher than everything else”
Listen at 36:44
Kimi had the lowest benchmark cost and duration among compared models.
“Kimi wins on both cost and duration”
Listen at 37:38
Nufar is considering using GPT more frequently in her primary model roster.
“I'm definitely contemplating reaching out to GPT more as part of my top two.”
Listen at 39:11
A well-designed benchmark can be reused after significant model shifts.
“you can run the same benchmark every time there is a significant shift that you're contemplating tapping into.”
Listen at 41:21
Advancing AI capabilities are raising expectations for acceptable model performance.
“the bar is getting higher.”
Listen at 43:06
Evaluation criteria and quality standards for AI models change frequently.
“the considerations of what makes you change your mind or what good looks like, those change quite frequently”
Listen at 43:11
Testing multiple effort settings can reveal whether configuration changes evaluation results.
“if you want to be extra diligent, you can always run several models with different effort setting and see if it changes the picture”
Listen at 43:59
Operational AI tasks are likely strongly influenced by the surrounding system setup.
“for operations, I would guess it's going to be most influenced by the overall setup”
Listen at 45:11
For basic tasks, capable open-weight models may perform indistinguishably from commercial models.
“for them, you will probably see that any decent model, let's call it Kimia and above or Mistrel and GLM and above, you will not be able to see any noticeable difference and then open source will be as good as it gets”
Listen at 46:11
Commercial models can retain an advantage on sophisticated tasks.
“For the more aggressive things or more sophisticated stuff, sometimes you will still see the gap between commercial and open source”
Listen at 46:23
Open-source models currently suffice for most knowledge-work tasks with a good harness.
“as of now, open source is good enough for most knowledge work tasks, especially if the harness in which it's working is a good one.”
Listen at 46:35
Agentic tools require measurable goals rather than vague quality requests.
“Agentic tools need goals. Saying create a good blog post is not measurable.”
Listen at 47:16
Specific constraints make agentic-tool goals more measurable and attainable.
“Saying create a blog post that does not include the following and is limited by these number of words and has the following parts is even better because it's highly measurable and highly attainable.”
Listen at 47:27
AI subscriptions may become less generous in the future.
“a good prediction to make is that it might not stay as generous in the future”
Listen at 48:29
AI subscriptions may become less generous over time.
“it might not stay as generous in the future.”
Listen at 48:31
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Weekly email
State of the Pod, every Monday
The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.