Jul 26, 2026 · 1h 34m
Anthropic product leader explains how AI is rewriting the product management playbook
Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future | Dianne Penn
As artificial intelligence advances, the product management discipline is undergoing a fundamental shift from writing static specifications to designing dynamic evaluation systems.
- 1Product managers in AI must replace traditional PRDs with robust evaluation sets to guide model behavior.
Don't miss
Dianne Penn explains why evals are the new PRDs and how PMs must transition to test-driven development to build successful AI products.
The brief
Anthropic's first technical product manager, Dianne Penn, reveals how building frontier AI models has fundamentally rewritten the rules of product management, shifting the focus from traditional roadmaps to rapid, evaluation-driven experimentation.
In frontier AI, product managers must trade traditional product requirement documents for robust evaluation sets. Success relies on sweating individual tokens as much as pixels, reading model transcripts deeply, and treating evals as the new software spec.
Penn details the strategy behind Anthropic Labs, which focuses on high-leverage, discontinuous bets like Claude Code and the Model Context Protocol, demonstrating how emerging capabilities are uncovered and refined directly through scaling.
Despite the blistering pace of shipping AI models, Penn argues that hard-earned human judgment, persistence, and deep hands-on tinkering remain irreplaceable, acting as the ultimate sparring partners to sharpen human thinking.
What was said on this episode
36 statements · 27 positive · 1 negative · 2 mixed · 6 neutral
Opus 4.5 and Claude Code mutually enabled their impact and adoption.
“Opus 4.5 wouldn't have had that moment without a product like cloud code. And cloud code wouldn't have had that type of adoption accelerated without Opus 4.5.”
Listen at 0:40
AI product teams should optimize token usage as seriously as visual design.
“You have to sweat the tokens as much as you sweat the pixels.”
Listen at 1:10
Improving Opus 3 for long-form coding helped Anthropic differentiate competitively.
“It ended up being a relatively smaller change from a training perspective, but it ended up helping us differentiate in the early days competitively for users”
Listen at 11:48
Frontier products are needed for users to experience frontier models’ value.
“you need Frontier products in order to have Frontier models and for people to feel the magic of Frontier models”
Listen at 12:59
Adaptability and agility are important as AI capabilities change rapidly.
“the adaptability of when you're faced with new information, how do you then make better decisions versus keeping the same plan? That agility is really important.”
Listen at 15:55
Scaling AI models can produce discontinuous jumps in emergent capabilities.
“as you add in more data and you train the models with more compute, you essentially see these actually discontinuous emergent capabilities jump”
Listen at 18:28
Without evaluations, teams may not detect emergent capability jumps.
“unless you have the evals, unless you have the systems to test these jumps might actually happen. And you don't know.”
Listen at 19:07
Current AI models have unexplored product and user potential.
“there's product overhang and user overhang to maybe put it in our PM language, even on today's models”
Listen at 19:27
Using AI models is necessary for developing strong product ideas.
“You have to be using the models to then come up with good, then great, then better ideas, and there's no substitutes for that.”
Listen at 21:16
AI experimentation benefits from communal discovery rather than isolated individual work.
“I think we could be doing more to actually bring that communal discovery when we do experimentation.”
Listen at 22:42
Anthropic Labs pursues large, discontinuous bets outside the core roadmap.
“The thesis of Labs in many ways is identifying and pulling the thread on the thread of discontinuous large bets that might not be in the core roadmap”
Listen at 23:57
Small Labs pods can move faster than large teams pursuing ambiguous ideas.
“the pods within labs is small. Sometimes these ideas start with one engineer. Right. And I think sometimes when there's almost really large teams pursuing very ambiguous large ideas, you end up actually being slowed down because of that.”
Listen at 26:33
Product teams should translate vague user feedback into specific failure trajectories.
“part of the time of the team is understanding, okay, what's the trajectory of why that user gave that feedback?”
Listen at 30:30
Successful Anthropic researchers tend to be strong first-principles thinkers.
“the most successful researchers and research leadership at Anthropic are folks who are really strong first principles thinkers about problems”
Listen at 32:23
AI product development should account for future model capabilities.
“how do you make sure what you're building is actually forward compatible?”
Listen at 34:30
AI safeguards and pre-release testing must rapidly evolve with frontier capabilities.
“as frontier models become more capable, the safeguards and the ways of red teaming and testing and the pre release process also needs to evolve and adapt quickly”
Listen at 36:17
Anthropic aims to make its AI systems broadly inclusive and accessible.
“Our goal is to be to develop these systems in the models to be as inclusive as possible.”
Listen at 37:58
Anthropic treats evaluations as a modern substitute for some PRDs.
“we actually have a saying on the team of evals are the new PRDs”
Listen at 41:41
AI product teams must scrutinize token behavior alongside interface design.
“The pixels here you have to sweat the tokens as much as you sweat the pixels.”
Listen at 42:50
Reliable schema and JSON output is fundamental to Claude’s agent capabilities.
“the early cloud models were not very good at following specific schemas. So things like outputs and JSON. And now that is fundamental to Claude being able to be a good agent.”
Listen at 44:36
Evaluations help product teams improve AI user experiences by making quality measurable.
“having things like evals actually is a way not just for folks working on models, but generally within product to get to better user experiences because you can't improve what you can't measure”
Listen at 47:16
PRDs help large groups align on product experience and goals.
“PRDs are great vehicles for getting a very large group of people aligned on a set of sources of truth about experience and set of goals.”
Listen at 48:14
Managers of AI product teams should personally ship with AI technology.
“in order to be good managers of teams and PMs working with this technology, you have to be really hands on yourself and have spent not just time tinkering, but actually shipping with this technology”
Listen at 50:36
AI career success favors people who experiment and ship hands-on.
“the folks that will be most successful regardless of their level are people who love working with AI and are exploring and experimenting and carving out the time not just for the experimentation, but actually hands on shipping end to end”
Listen at 52:50
Deep focus on one or two AI tools is preferable to shallow experimentation.
“my lens has been, how do I go deep in one to two of them myself?”
Listen at 56:26
Claude can help users prepare for difficult workplace conversations.
“I use it a lot and actually prepping for how to have better conversations in the moment, during crucial conversations.”
Listen at 59:13
Claude should augment rather than replace a user’s thinking.
“what I want to make sure, and maybe this is what you're describing, is CLAW doesn't take over all of my thinking for me”
Listen at 1:01:50
Claude’s pushback can improve users’ thinking and ideas.
“sometimes it's having Claude push back makes me better”
Listen at 1:03:40
Claude becomes more useful when it knows when to challenge users.
“in order for Claude to be more useful, the general approach has to be that it knows when to push back”
Listen at 1:05:55
AI capabilities remain uneven across different tasks.
“the technology is jagged edged”
Listen at 1:09:35
Improving Claude’s writing may shift attention toward greater proactivity.
“once we improve, let's say, writing and tone and character, we probably will say, how do we have Claude be even more proactive?”
Listen at 1:11:17
Human judgment will remain critical for product leaders.
“that hard earned judgment is an area for product leaders and just generally will continue to be really critical”
Listen at 1:12:33
AI has substantially transformed software engineering.
“software engineering has been really transformed by AI”
Listen at 1:13:31
Curiosity, persistence, and independent judgment will matter in the future.
“curiosity for learning, persistence, believing in your own inner voice, developing and then believing in your own inner voice”
Listen at 1:14:33
Product managers remain necessary despite increasingly capable AI models.
“there is this question in the community of do we still need PMs when the models are so capable”
Listen at 1:23:01
User feedback about Claude’s failures helps Anthropic improve it.
“pushing Claude, telling us where it's falling down, those help us make cloud better”
Listen at 1:32:15
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.