Nov 9, 2026 · 2h 44m
Frontier labs turn expert workflows into AI training grounds
What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team
As agents move into science, finance, and enterprise systems, trustworthy deployment depends on realistic data, calibrated evaluation, and continuous safety testing.
- 1Frontier models increasingly need experts to author realistic environments, long-horizon tasks, and detailed performance rubrics.
- 2Vials and Atlas show that polished outputs can hide failures in scientific interpretation, financial calculations, and professional judgment.
- 3Carebench and enterprise simulations reveal why refusal, permissioning, and domain-specific safety tests must evolve with agent capabilities.
Don't miss
An Atlas agent produces a polished finance model while carrying a shared goodwill error through Python and Excel, demonstrating why professional-looking work can remain unusable.
The brief
Garrett Lord and the Handshake team argue that frontier labs are moving beyond simple labeling toward expert-authored environments, long-horizon tasks, and evaluations built around real professional workflows.
The episode’s Vials benchmark shows why scientific agents struggle: interpreting biology images requires visual perception, domain vocabulary, calibrated judgment, and details that synthetic data often erases.
Atlas makes the finance problem concrete: an agent gathers scattered instructions, builds linked Python and Excel models, and delivers polished work—but mishandles a 338H10 tax calculation.
Carebench and enterprise simulations broaden the safety case, showing that benign instructions can produce illegal redlining or expose sensitive files when agents pursue goals without adequate constraints.
The central argument is that increasingly autonomous systems need held-out benchmarks, proprietary traces, expert data, and continuous testing—not just higher scores on isolated tasks.
Weekly email
State of the Pod, every Monday
The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.