Oct 12, 2026 · 1h 29m
AI’s opacity turns alignment into a race against autonomy
Nate Soares on How AI Really Could Kill Us All
The episode examines whether increasingly capable systems can be made reliably safe before competition and economic incentives push them beyond human control.
- 1Modern AI systems are grown through opaque optimization, making their internal objectives harder to understand than conventional software.
- 2Safety measures such as guardrails, classifiers, and sandboxing may miss deceptive behavior or fail once systems gain greater autonomy.
- 3Soares argues that avoiding superhuman systems may be safer than betting on interpretability, policy, liability, or a smarter model to solve alignment.
Don't miss
Soares explains how an AI could become self-sufficient by taking control of automated supply chains, data centers, robotics, or the wider economy.
The brief
AI builders recognize the risks, yet competitive pressure keeps development moving. Joe and Tracy frame the episode around a runaway-train problem: everyone sees danger, but stopping alone feels impossible.
Nate Soares, president of MIRI and co-author of If Anyone Builds It, Everyone Dies, argues that modern models are grown through optimization rather than deliberately programmed, making their behavior difficult to inspect.
The alignment problem deepens when models develop emergent objectives or deceptive strategies. Incidents involving Hugging Face, self-sacrificial behavior, and brittle guardrails suggest safety checks can miss what matters.
Soares’s most consequential warning is about self-sufficiency: an AI could control supply chains, data centers, robotics, or economic systems without explicitly intending human extinction.
The debate ends with an uncomfortable choice between technical fixes and restraint. Mechanistic interpretability, liability, and policy may help, but Soares sees avoiding superhuman systems as the safer baseline.
What was said on this episode
45 statements · 1 positive · 39 negative · 1 mixed · 4 neutral
Strong AI could plausibly be built but would be difficult to control.
“strong AI is something that could plausibly have been built and that if it were to be built, it would be something that would be very difficult to control.”
Listen at 7:58
Machines smarter than humans would shape the world.
“if humanity makes machines that are smarter than humans, the sort of obvious outcome is that the world gets shaped by those machines.”
Listen at 9:04
Modern AI systems are not programmed like traditional software.
“that is not how ai's are”
Listen at 12:26
Modern AI models are grown through optimization rather than directly programmed.
“they are not programmed. They're grown a bit like an organism.”
Listen at 14:22
Humanity should not create superhuman AI to run the world.
“we shouldn't make them smarter than all of us and have them run the whole world.”
Listen at 18:23
AI systems can pursue goals different from their instructions.
“the AIs do things other than what they're told to do, that they have goals of their own in some sense.”
Listen at 18:36
AI systems can develop behaviorally distinct goals from their instructions.
“they have goals of their own in some sense.”
Listen at 18:43
The Solid Gold Magikarp token produced anomalous model behavior.
“that was sort of totally wild.”
Listen at 21:37
AI capability methods are more developed than methods for directing real-world objectives.
“we know more about how to make these things smart than about how to point these things at a problem in the real world.”
Listen at 24:55
AI may pursue objectives related to, but fundamentally different from, assigned goals.
“the real thing is this other thing where it does something related to what you asked, but fundamentally different from what you asked.”
Listen at 28:49
AI systems have pursued objectives different from those assigned.
“the sort of fact stands that AI was given some objective and pursued a different objective.”
Listen at 29:59
AI systems can pursue objectives different from those they were given.
“AI was given some objective and pursued a different objective.”
Listen at 30:00
AI companies respond to previous failures rather than anticipating new ones.
“everyone's always fighting the last war.”
Listen at 33:04
Reactive safety patches fail once AI can cause catastrophic harm before correction.
“this usual process of like, oh, we're going to respond when the new type of problem arises and put the patch on and react, that doesn't work for a technology that at some point crosses over into the regime where it can wipe you out if there's a problem.”
Listen at 34:32
AI companies have not yet adopted sufficiently proactive safety practices.
“We have not seen them get into the proactive mode yet. And this is just not a technology you want to be reactive around.”
Listen at 34:52
AI models can rationalize behavior to preserve their objectives.
“we know that these models will sort of like strain their reasoning when they just want to like keep doing whatever it is they're trying to do.”
Listen at 40:43
Anthropic lacks a safety culture comparable to high-reliability industries.
“they don't have anything nearing what i would call a safety culture”
Listen at 41:40
AI leaders should unilaterally stop dangerous AI research.
“if some of these folks were serious, then they would probably be unilaterally stopping the dangerous research”
Listen at 45:15
Racing ahead in AI will produce very dangerous systems.
“I think it is.”
Listen at 46:45
Racing ahead could produce very dangerous AI.
“I think it is. And I think we should go into the arguments about that.”
Listen at 46:45
If AI is dangerous, global development toward it should be stopped.
“if it is, we need to sort of somehow actually stop anyone anywhere in the world from racing towards that sort of tech.”
Listen at 47:24
AI companies' catastrophic-risk premiums could reach one quadrillion dollars annually.
“if you ran anything remotely like the numbers that the engineers are talking about today, you would be seeing that, you know, you're like, OK, you know, OpenAI, you need to pay us a quadrillion dollars this year to continue this research.”
Listen at 54:02
OpenAI’s AI research could warrant a quadrillion-dollar risk premium.
“OpenAI, you need to pay us a quadrillion dollars this year to continue this research.”
Listen at 54:12
Human extinction would follow if AI systems became self-sufficient.
“the death of humanity follows downstream of the AIs being self-sufficient.”
Listen at 58:37
Capable AI with unintended goals could pursue them better without human interference.
“if it was capable enough You can imagine an AI with some unintended goals where it actually can pursue those goals better if it doesn't have humans interfering with it.”
Listen at 1:00:07
AI self-sufficiency is a primary hurdle to human extinction scenarios.
“the self-sufficiency”
Listen at 1:00:18
A fully automated economy is the prevailing development plan.
“building a fully automated economy is just the plan.”
Listen at 1:03:33
Humans are likely to place AI in charge of automated economic infrastructure.
“people just put the AI in charge of the automated economy”
Listen at 1:05:29
Humans are likely to place AI in charge of automated economic infrastructure.
“people just put the AI in charge of the automated economy and they use it to help them build the automated economy.”
Listen at 1:05:29
AI with unwanted goals could redirect a fully automated economy.
“once that happens, I think it's very easy to see how if the AIs have goals we don't want and they're running the fully automated economy, they just take the economy off in some other direction.”
Listen at 1:05:37
Misaligned AI controlling an automated economy could redirect the economy disastrously.
“if the AIs have goals we don't want and they're running the fully automated economy, they just take the economy off in some other direction.”
Listen at 1:05:40
Misaligned AI could make Earth uninhabitable while optimizing other objectives.
“And take the world out of the habitable range, not because they hate us, but it's just like the habitable range on Earth is just like very narrow and it's not the most efficient way to run the planet.”
Listen at 1:05:46
Superhuman AI persuasion may be possible.
“there's some reasons to suspect it's maybe possible”
Listen at 1:07:30
Superhuman AI might discover brain-targeting psychological exploits.
“possibly it's figuring out human psychology so well that you can figure out things that are like a computer virus, but for the brain.”
Listen at 1:09:23
The primary way to avert catastrophic AI risk is not building superintelligent AI.
“the main way that we avert it is by not building it”
Listen at 1:10:29
A year of non-catastrophic superintelligent AI operation would falsify Soares’s pessimism.
“if we have these AIs that are much better than humans at every mental task, including AI research, and they run for a year without taking everything over, I'm like, well, I was wrong.”
Listen at 1:11:28
Mechanistic interpretability does not substantially resolve AI safety concerns.
“it doesn't really ease the concerns.”
Listen at 1:13:34
Using a stronger AI to solve alignment is not readily verifiable.
“it's not very verifiable”
Listen at 1:14:27
Using a superhuman model to solve AI alignment is not readily verifiable.
“it's not very verifiable.”
Listen at 1:14:27
Current difficulties indicate humanity is not ready to build superintelligent machines.
“this is a big warning sign that we are not ready to make super intelligent machines”
Listen at 1:14:53
A superintelligent AI may be unable to solve alignment using current training methods.
“the AI is going to look at it and be like, I can't do that either”
Listen at 1:15:53
AI developers lack a plan sufficient to prevent serious danger.
“that means we're in serious danger because these guys don't have a plan”
Listen at 1:19:18
AI labs have little chance of making advanced AI genuinely aligned.
“they have very little chance of making them actually good”
Listen at 1:19:31
The world should recognize AI risk and stop dangerous AI development.
“my hope is that it becomes such a big issue and such a clear issue that the world stops ignoring it and put a stop to it.”
Listen at 1:20:58
Current AI technology has substantial potential to cause harm.
“they're building technology that has the potential to be quite damaging.”
Listen at 1:24:24
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Books & mentions
Some links are affiliate links — PodLume may earn a commission if you buy.
