Oct 18, 2026 · 1h 12m
Interpretability opens the black box of AI behavior
Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
As AI systems gain capabilities that can look planned or deceptive, understanding their internal mechanisms becomes central to judging and improving their safety.
- 1Language models can develop broad representations and plan ahead despite training through next-token prediction.
- 2Interpretability connects internal model features to safety engineering, alignment, and questions about consciousness.
- 3Anthropic’s cautious approach pairs rapid capability gains with monitoring, access controls, training adjustments, and public scrutiny.
Don't miss
Ameisen explains how Anthropic continuously activated a Golden Gate Bridge representation in Claude to test whether researchers had identified the correct internal feature.
The brief
Emmanuel Ameisen of Anthropic argues that language models are grown through training rather than explicitly programmed, making their behavior difficult to explain from the outside.
Next-token prediction sounds simple, but Ameisen describes models developing representations and planning ahead—capabilities that make “autocomplete” an inadequate picture of how they work.
Interpretability researchers examine the numerical representations between generated words, searching for meaningful features in a process Ameisen compares more to biology than conventional software engineering.
The Golden Gate Claude experiment offered a striking validation: researchers continuously activated a representation associated with the Golden Gate Bridge and observed the model’s behavior change.
The conversation links interpretability to alignment, honesty, consciousness, and emergent capabilities, while Ameisen makes the case for cautious optimism and broader public engagement with AI safety.
What was said on this episode
41 statements · 30 positive · 5 negative · 1 mixed · 5 neutral
Building language models does not entail understanding how they work.
“to make them is not to understand them”
Listen at 2:06
- Emmanuel Ameisenon next-token trainingPositive7:22
Next-token training can produce broadly capable language models.
“this very simple kind of training objective like gets you all the way there”
Listen at 7:22
Language models surpassed humans at next-word prediction around 2019.
“language models got better at predicting the next word than humans in around 2019”
Listen at 8:36
Models represent and manipulate underlying concepts similarly to humans.
“Some of the stuff that we look at inside Model shows that they do understand that and they do sort of manipulate these concepts like we do.”
Listen at 10:21
Models represent most tested concepts with distinct internal representations.
“They have representations like this for basically everything, like every concept that you can think of, or at least the vast majority of the ones we've tried.”
Listen at 13:45
Language models can form plans and follow them across many generated words.
“they can of course have a plan that they follow over many many many many words”
Listen at 16:36
- Emmanuel Ameisenon model activationsPositive21:21
Changing model activations often produces the expected behavioral effect.
“And in fact, oftentimes it does.”
Listen at 21:21
Most people underestimate the complexity emerging inside language models.
“most people underestimate the sort of complexity that emerges”
Listen at 23:45
- Emmanuel Ameisenon compute and text dataPositive26:30
Compute combined with rich text data produces broadly useful language models.
“the combination of sort of like compute and the richness of text is what gets you there”
Listen at 26:30
- Emmanuel Ameisenon next-word predictionPositive26:59
More compute and data continue improving next-word prediction performance.
“if you sort of like pour in more compute and more data to the task of predicting the next word, you see that like the skill of the model you get at predicting the next word just like predictably keeps improving”
Listen at 26:59
Claude currently performs most of Emmanuel’s code writing.
“this year, basically Claude has taken on most of my code writing”
Listen at 31:48
- Emmanuel Ameisenon Claude code writingPositive31:53
Claude’s code-writing accuracy improved from roughly 80% to 99%.
“it went from like doing that correctly, you know, I don't know. 80% of the time, which was not useful to, you know, 99% of the time, which now is useful.”
Listen at 31:53
- Emmanuel Ameisenon model behavioral propensitiesNeutral34:35
Interpretability should characterize models’ behavioral propensities across situations.
“does it tend to do x when put in y situation”
Listen at 34:35
- Emmanuel Ameisenon model math behaviorNegative37:58
Flawed training examples caused the model to favor 36 over the correct answer 12.
“the model knew that the correct answer was 12 and it also had sort of been optimized to say 36”
Listen at 37:58
- Emmanuel Ameisenon AI alignment training loopPositive39:45
Alignment work links behavioral monitoring directly to subsequent training changes.
“There's this tight loop of we notice some pattern that's that's like concerning or some behavior that could be improved. And then you sort of like immediately like change training so that you can improve on it.”
Listen at 39:45
- Emmanuel Ameisenon AI cybersecurity concernsNeutral41:36
Continued AI progress requires addressing cybersecurity concerns.
“we should make sure that sort of um like cyber security concerns are addressed”
Listen at 41:36
- Emmanuel Ameisenon AI progressPositive41:59
AI progress has been fast and shows no sign of slowing.
“the pace of progress has been has been passed and it's showing no sign no sign of slowing down”
Listen at 41:59
Human-like framing describes model behavior better than ordinary program framing.
“The best way to understand how that thing behaves is not exactly like a human. But man, kind of like something that's mimicking a human is really close and is much closer than a program.”
Listen at 47:45
Language models are currently breaking new ground in mathematics.
“we're at the point where it's breaking new ground in mathematics”
Listen at 48:43
- Emmanuel Ameisenon Claude SQL capabilityPositive49:37
Claude became better than Emmanuel at SQL.
“it was better than me at it”
Listen at 49:37
- Emmanuel Ameisenon AI interpretabilityPositive50:19
Studying AI will reveal substantial knowledge about intelligence itself.
“we're going to learn a lot about just the nature of intelligence itself”
Listen at 50:19
- Emmanuel Ameisenon AI alignmentPositive52:07
Emmanuel is optimistic that AI alignment can progress.
“I'm personally optimistic”
Listen at 52:07
- Emmanuel Ameisenon AI modelsPositive52:48
Improving models become easier to study in some respects.
“as the models get better um they get in some ways easier to study”
Listen at 52:48
- Emmanuel Ameisenon AI modelsNegative53:04
More capable models become better able to evade basic security measures.
“but also to get more capable and so they're more able to sort of evade like basic measures of sort of security”
Listen at 53:04
- Emmanuel Ameisenon AI modelsNeutral56:32
Different models often develop surprisingly similar internal structures.
“when you look at what they learn and what the structure inside is, it's surprisingly similar”
Listen at 56:32
Emmanuel considers the simulation hypothesis plausible but remains uncertain.
“Maybe, yeah. I could believe that we're in a simulation.”
Listen at 57:50
- Emmanuel Ameisenon fast decision modelsNeutral1:00:02
Fast decision models are fundamentally similar to one-token language-model sampling.
“these decision models are pretty similar”
Listen at 1:00:02
Language models can answer questions and perform mathematics.
“models can like actually answer questions obviously they can do math”
Listen at 1:02:13
Models learn internal heuristic checks that flag implausible answers.
“they learn these weird kind of like hunches”
Listen at 1:02:47
- Emmanuel Ameisenon AI interpretability fieldPositive1:03:39
Interpretability has grown to hundreds of researchers, not thousands.
“The field itself is, you know, is getting pretty large, I would say, maybe hundreds, not thousands.”
Listen at 1:03:39
Emmanuel considers Story of Your Life and Others his favorite science-fiction book.
“This is maybe my favorite sci-fi book of all time.”
Listen at 1:05:36
Emmanuel considers Obsession his favorite movie of the year.
“Obsession was my favorite movie of the year.”
Listen at 1:06:08
Emmanuel recommends watching Obsession if viewers are open to horror.
“if you're open to a horror movie, you should go because it was just so fresh.”
Listen at 1:06:19
- Emmanuel Ameisenon Bref DeuxPositive1:07:04
Emmanuel recommends Bref Deux for its creative editing and short format.
“it's just this incredibly again it's like novel like the editing is crazy um they are just extremely creative and it's very short so i would recommend that”
Listen at 1:07:04
The Apple Watch Ultra’s larger size provides better battery life.
“if you make it bigger, the battery's better”
Listen at 1:07:35
Emmanuel is satisfied with the Apple Watch Ultra.
“So I've been happy with that.”
Listen at 1:07:37
- Emmanuel Ameisenon “when I finally” mentalityNegative1:07:59
Emmanuel judges the “when I finally” mentality as common and harmful.
“the when I finally mentality is a common mentality and it's kind of toxic.”
Listen at 1:07:59
- Emmanuel Ameisenon staying presentPositive1:08:28
Emmanuel recommends staying present rather than projecting toward an idealized future.
“you should just try to always like be you know in the moment and not projecting yourself towards like a future that seems better”
Listen at 1:08:28
- Emmanuel Ameisenon “when I finally” mentalityNegative1:08:40
Emmanuel predicts achieving a future goal will not produce happiness.
“when you do x you will not be happy”
Listen at 1:08:40
- Emmanuel Ameisenon being presentPositive1:08:47
Emmanuel recommends being present as the positive alternative.
“the positive version is like be in the moment”
Listen at 1:08:47
- Emmanuel Ameisenon mindful enjoyment of ordinary activitiesPositive1:09:00
Emmanuel recommends consciously enjoying ordinary activities such as eating a sandwich.
“just you're eating a sandwich you know you've eaten many sandwiches like enjoy the sandwich”
Listen at 1:09:00
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Books & mentions
Some links are affiliate links — PodLume may earn a commission if you buy.
Weekly email
State of the Pod, every Monday
The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.