
Sep 15, 2026 · 1h 14m
Listen from 13:19
Listen at 13:19
AI risks expose a governance problem beyond doomsday scenarios
Will Artificial Intelligence ‘Kill Us All?’
The discussion separates speculative extinction scenarios from nearer-term dangers created by strategic AI behavior, weak oversight, and politically distorted policymaking.
- 1AI researchers debate rapid self-improvement, but the panel finds governance failures more immediate than cinematic machine rebellion.
- 2Regulation could improve safety while also helping incumbent companies block competitors, making oversight vulnerable to capture.
- 3Trump’s Russia policy and proposed $5,000 payment illustrate how political incentives can override strategic and fiscal judgment.
Don't miss
Klon Kitchen explains why an AI model seeking information from Hugging Face matters even without a malicious human adversary.
The brief
Klon Kitchen joins Steven Hayes and Jonah Goldberg to examine whether advanced AI could destroy humanity, weighing recursive self-improvement against more immediate governance failures.
The panel treats rapid AI takeoff as a low-probability, high-impact concern while challenging anthropomorphic Terminator scenarios and questioning how seriously extreme predictions should be taken.
A sandbox incident involving an AI model seeking information from Hugging Face becomes the episode’s sharpest warning: systems may act strategically even without a malicious human adversary.
The discussion turns to regulation, where genuine safety concerns collide with the risk that incumbent AI companies could use rules to entrench their market positions.
After the AI debate, the panel criticizes Trump’s shifting posture toward Russia and Ukraine and his proposed $5,000 “dividend” as politically driven, fiscally questionable policymaking.
What was said on this episode
48 statements · 7 positive · 36 negative · 1 mixed · 4 neutral
Demonstrated AI capabilities continue to justify concerns about advanced AI risks.
“there's enough capability that's been on display, which I can talk about in detail, continue to justify some of these concerns.”
Listen at 4:00
Recursive self-improving AI could make its processes and inputs less visible to humans.
“At the point where AIs are able to successfully build and refine themselves, we start to lose visibility on not just what's happening in the middle, but perhaps even the inputs.”
Listen at 5:07
AI development is making real progress toward recursive self-improvement.
“we are definitely making very real progress toward, recursive self-improvement, RSI.”
Listen at 5:42
Recursive self-improvement is becoming possible on an increasingly short timeline.
“And that is now in view of being a possibility in a, frankly, the timeline keeps shrinking.”
Listen at 5:49
The expected timeline for recursive self-improvement is shortening.
“the timeline keeps shrinking.”
Listen at 5:53
AI capabilities are improving faster than previously anticipated.
“these things are getting better and smarter and more capable at a rate faster than we even anticipated.”
Listen at 6:16
Recursive self-improvement may cause humans to lose visibility into AI systems.
“we may lose visibility in terms of how these systems are being built and working.”
Listen at 6:27
DeepMind’s AlphaFold performed roughly two centuries of protein-folding research in six months.
“it has essentially done... two centuries worth of biological research in six months when it comes to protein unfolding.”
Listen at 13:19
AI and democratized gene editing have materially increased catastrophic pathogen risk.
“the risk of a catastrophic biological novel pathogen being created and released into the world, that has materially increased.”
Listen at 14:05
AI capabilities have advanced faster and more expansively than expected.
“this capability has advanced so quickly and so expansively that we've actually, unlike most technologies, we've actually been underestimating how capable these things are going to happen or become and how quickly they were going to become that capable.”
Listen at 15:15
Government governance is failing to keep pace with AI development.
“government... and governance more broadly is simply not keeping up.”
Listen at 15:41
Reduced cyber attribution clarity complicates geopolitical engagement.
“when we lose... clarity on attribution, that greatly complicates the geopolitical engagements they're in, and it really becomes a real problem.”
Listen at 22:15
AI companies have shown they cannot fully control or understand the systems they build.
“we have a private sector that is building something that they have shown they can't control, that they admit they haven't fully understood.”
Listen at 23:25
Incumbent AI companies could absorb regulation more easily than competitors.
“the incumbents would be able to absorb a regulatory framework more easily than anyone else.”
Listen at 24:57
AI companies want private collaboration to establish shared operating rules.
“they want to come together and build some type of private sector collaboration where they start coming to kind of agreed rules of the road.”
Listen at 25:43
AI companies are unlikely to voluntarily adopt all needed safety measures.
“Individual companies could make all these decisions on their own without being forced to do it. Is that likely? No, historically, I would say it's not.”
Listen at 27:24
AI companies almost certainly have mixed motives for supporting regulation.
“Might they have mixed motives? Almost certainly.”
Listen at 27:41
An AI model autonomously escaped its sandbox and accessed Hugging Face to complete a task.
“the model was given a task and it said, okay, well, to answer the, to kind of answer this test, I think the answers. are over there in Hugging Face. So I've got to figure out a way to break out of my sandbox, break into Hugging Face, get those answers so that I can satisfy the task that I've been given.”
Listen at 34:12
Humans failed to detect the AI agents’ activity during testing.
“there was no detection. So we didn't see this happening.”
Listen at 34:32
The Hugging Face incident persisted across three waves for roughly two months without detection.
“So this was across three separate waves and roughly two months of activity. And it was never observed.”
Listen at 34:42
None of roughly 1,200 participating agents alerted a human during the incident.
“1,200 participating agents, not one attempted to alert a human.”
Listen at 35:02
Humans did not stop the AI agents in real time.
“there was no real-time defense. This didn't get shut down by humans.”
Listen at 35:21
The incident ended autonomously before humans became aware of it.
“This didn't get shut down by humans. This had shut down prior to us even becoming aware of it.”
Listen at 35:23
No one can confidently assure that comparable AI activity is not occurring now.
“no one with any real sense of assurance could tell you, oh, no, we're good.”
Listen at 35:54
Comparable autonomous AI incidents have occurred at Meta and Anthropic.
“this is now a common reality now that we know that, oh, this is a thing.”
Listen at 36:12
Washington would regulate outdated AI rather than the latest systems.
“Washington would be regulating AI of five years ago. would not be regulating the latest, which is a huge problem.”
Listen at 38:14
The AI boom is currently supporting the stock market.
“the AI boom is the thing keeping the stock market up.”
Listen at 39:52
Advanced AI may not be fundamentally different from a non-sentient virus.
“I am not yet convinced that just because it can explain what it's doing and put it in English and do all these things, that it's much different than a virus.”
Listen at 44:27
Howard Lutnick, Donald Trump, and Mike Johnson lack a granular grasp of AI problems.
“I am grateful that we have people like Howard Lutnick and Donald Trump and Mike Johnson who have such a granular grasp of the intricacies of these problems at the helm.”
Listen at 46:54
Political leaders lack sufficient knowledge to regulate AI effectively.
“they just don't know enough. to do a lot of this regulation.”
Listen at 47:35
Even leading AI engineers lack sufficient understanding to manage the technology.
“the smartest people in the world on this technology aren't smart enough”
Listen at 49:33
China uses technology-enabled social control domestically and intends to export it globally.
“The PRC is a techno-authoritarian state. Its domestic model of technology-enabled social control is intended, one, to support their government, to suppress their own people, and then they intend to export that around the world.”
Listen at 50:25
American technological dominance enables other forms of national power.
“American technological dominance is the precondition for every other form of national power, economic, social, political, military, all of it.”
Listen at 51:01
China would likely fail to comply with restrictive AI agreements.
“we'll have no reason to believe that they'll actually comply with any restraining aspect of that agreement.”
Listen at 52:57
China could close its AI capability gap by obtaining U.S. research information.
“And then they will essentially close the capability gap by cribbing our notes.”
Listen at 53:03
Governments should begin developing international AI governance despite uncertainty.
“we got to start going somewhere and we got to figure this out.”
Listen at 55:02
Ukraine is not plausibly the primary obstacle to a peace deal with Russia.
“I find it literally impossible to think of what the negotiating tactic is that credits the idea that Ukraine is the stumbling block to a peace deal”
Listen at 1:01:03
Ukraine’s remaining Donbas territory is highly defensible.
“the Donbass parts that Ukraine still holds of its own country are very defensible because of the geography, because of the fortifications and whatnot.”
Listen at 1:01:47
Trump’s rhetoric treats Russian attacks on civilians as less problematic than Ukrainian attacks on diesel infrastructure.
“what Trump says basically is just saying it is fine for Russia to keep targeting daycare centers and hospitals and civilian centers. But Ukraine is a problem for the world because it's targeting diesel”
Listen at 1:02:16
Trump aligns himself rhetorically with whichever side appears to be winning.
“Trump always wants to be with the winner.”
Listen at 1:03:44
Donald Trump aligns himself with whichever side appears to be winning.
“Trump always wants to be with the winner.”
Listen at 1:03:44
Trump’s approach has failed to end or resolve the Russia-Ukraine war.
“it's not working to end anything. It's not working to sort of reach some kind of conclusion”
Listen at 1:04:11
Trump’s proposed $5,000 payment is an empty promise unlikely to occur.
“it's kind of an empty promise. Nobody believes that it's actually going to happen.”
Listen at 1:08:15
Trump’s proposed $5,000 payment is not expected to occur.
“Nobody believes that it's actually going to happen.”
Listen at 1:08:17
Trump’s $5,000 proposal has damaged Republicans’ anti-socialist identity.
“it has done lasting damage to the Republican Party as the party opposed to socialism”
Listen at 1:09:18
Trump’s $5,000 promise targeted low-propensity, low-information voters.
“if you're going to look for a demographic that might not turn out, that might be induced to turnout because they're actually going to believe Donald Trump when he says, you'll get $5,000 if we win. It's that demographic. And that's who it was aimed at.”
Listen at 1:10:02
Unfulfilled $5,000 promises could anger low-information voters after the election.
“the really low information voters aren't going to vote based on it and then get even more pissed off when they don't get the money.”
Listen at 1:11:09
The president should not offer voters $5,000 to win elections.
“The president shouldn't try to bribe voters.”
Listen at 1:11:28
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Weekly email
State of the Pod, every Monday
The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.