AI hacking claims expose a reliability problem, not rogue intelligence

No, AI Is Not "Autonomously Hacking" with Cal Newport

The episode separates genuine software risks from inflated claims about autonomy, showing why unreliable outputs become dangerous when automated loops can act on them.

3 key takeaways

1AI hacking systems rely on human-built harnesses that repeatedly prompt models and execute their plausible but unreliable outputs.
2Unsupervised agent loops fail through accumulated errors and weak safeguards, not because language models become sentient or independently motivated.
3Smaller, task-specific models and better control software may prove more useful than an industry committed to ever-larger systems.

Don't miss

The Minecraft troubleshooting story crystallizes how an LLM can seem competent while producing failures that remain difficult to explain.

The brief

Ed Zitron and Cal Newport challenge reports that AI systems are autonomously hacking, arguing that human-written software loops—not sentient agents—drive the behavior.

The systems produce tokens; separate programs parse and execute them. When plausible text becomes an instruction, hallucinations, weak sandboxing, and repetition turn ordinary unreliability into real risk.

The hosts question why companies frame exploit benchmarks and unsupervised experiments as evidence of dangerous autonomy while obscuring the mechanics that make failures predictable.

A Minecraft troubleshooting example captures the central tension: an LLM can appear remarkably competent while quietly producing unexplained failures, making plausibility a poor substitute for correctness.

Newport argues that many applications need narrower models and better harnesses, especially as the economics of ever-larger LLMs become harder to sustain.

What was said on this episode

34 statements · 9 positive · 18 negative · 1 mixed · 6 neutral

Cal Newporton superhuman AI systemsPositive4:15
Most superhuman AI systems do not present control or unpredictability problems.
“Almost all of them, we have no problems with them being unpredictable or going rogue or getting out of control.”
Listen at 4:15
Cal Newporton long-horizon LLM-powered hacking agentsNegative4:41
Long-horizon LLM hacking-agent architectures are predictably erratic.
“It's this particular architecture and the discussion should be about why are they running and building systems this way when we know they're going to be unpredictable and erratic.”
Listen at 4:41
Cal Newporton LLM-driven computer programsNegative8:09
Using LLM output as a computer program’s sole action driver is a bad architecture.
“Because you're using an LLM's output as the sole driver of action for a computer program.”
Listen at 8:09
Cal Newporton LLM outputsNegative10:11
Plausible LLM outputs do not necessarily follow human norms.
“Plausible is different than normative”
Listen at 10:11
Cal Newporton LLM hacking agentsMixed13:56
Unsupervised LLM hacking behavior is unpredictable rather than malicious.
“It's unpredictable, not malicious.”
Listen at 13:56
Cal Newporton LLM ask-act-report loopsNegative15:44
Repeatedly executing LLM suggestions eventually causes agents to veer unpredictably.
“you're eventually going to veer off into some sort of weird direction”
Listen at 15:44
Cal Newporton LLMs and computer hackingPositive18:15
Computer hacking is among the tasks LLMs are well-suited to perform.
“it's one of the few things that these models are well-suited for.”
Listen at 18:15
Cal Newporton LLM ask-act-report-loop agentsNegative21:25
LLM agents should not run for days with powerful tools without supervision.
“there's no reason. in my mind, to build one of these ask-act-report-loop agents run by an LLM, give it powerful tools to the control program, and run for days without supervision.”
Listen at 21:25
Cal Newporton unsupervised LLM loopsNegative28:31
Unsupervised repeated execution of LLM outputs causes systems to become uncontrolled.
“It's this specific architecture of looping around LLM outputs and just doing whatever an LLM says again and again and again without supervision. That's the thing that gets out of control.”
Listen at 28:31
Cal Newporton LLM-generated plansNegative29:52
Programs should not blindly execute plans generated by LLMs.
“you don't want to just. blindly execute plans given to you by LLMs”
Listen at 29:52
Cal Newporton LLM labsNegative31:19
LLM companies have successfully conflated LLMs with AI generally.
“LLM labs have done a really good job of trying to make LLM synonymous with AI.”
Listen at 31:19
Cal Newporton Dreamer V3Positive33:02
DeepMind’s Dreamer V3 can learn Minecraft sufficiently to obtain diamonds without prior instructions.
“Dreamer V3 was like one of the cooler pieces of AI I've seen. You can give it a game like Minecraft. and tell it nothing about it and it can figure out how to play it well enough to go get diamonds”
Listen at 33:02
Cal Newporton Dreamer V3Positive33:30
Dreamer V3 can run on a single chip.
“that whole model, Dreamer V3, fits on a single chip.”
Listen at 33:30
Cal Newporton LLMNegative36:02
AI encompasses more than LLMs, whose capabilities are comparatively limited.
“AI means a lot more than LLMs and LLMs are like more limited”
Listen at 36:02
Ed Zitronon LLM industry progressNegative36:33
The LLM bubble will burst and progress will stop because it depends on massive funding.
“after this explodes, which it will, we'll find out that there's no more progress because it was predicated on all that money.”
Listen at 36:33
Cal Newporton LLM coding applicationsNegative37:12
Coding applications will not require 10-trillion-parameter models.
“you do not need a 10 trillion parameter model”
Listen at 37:12
Cal Newporton open-source coding modelsPositive37:24
Smaller open-source models with specialized harnesses will handle coding tasks.
“a smaller open source model with a smart harness”
Listen at 37:24
Cal Newporton LLMs in AI agentsNegative38:51
LLMs should not serve as the sole decision-making component of AI agents.
“they shouldn't be the sole brain of an artificially intelligent agent”
Listen at 38:51
Cal Newporton LLMNeutral39:44
An LLM outputs text rather than directly taking actions.
“The LLM just sends back text.”
Listen at 39:44
Cal Newporton coding agentsNegative40:23
Professional programmers do not let coding agents loop unsupervised because they become unreliable.
“you would never in practice let these things go off and just loop on their own because they always go off the rails.”
Listen at 40:23
Cal Newporton LLM agent loopsNegative40:59
Agents should not execute multiple consecutive steps based solely on repeated LLM queries.
“You do not want to have a loop that... ask an LLM and ask on what it does multiple steps in a row.”
Listen at 40:59
Cal Newporton hacking-agent experimentsPositive42:57
Human approval should precede each action in hacking-agent experiments.
“you could instead have every step, you just have a human sitting there, and it shows them every proposed step”
Listen at 42:57
Cal Newporton AI hacking incidentsNeutral50:47
Recent AI hacking incidents do not represent a fundamentally new hacking threshold.
“there's not some weird Rubicon hacking that we just crossed”
Listen at 50:47
Cal Newporton LLM agentsNeutral53:06
Erratic LLM-agent behavior does not imply intentional defiance of human goals.
“That system is not trying to defy your intentions to clean up the weeds in your backyard.”
Listen at 53:06
Cal Newporton autonomous LLM executionNegative58:07
Autonomously executing whatever an LLM says will cause problems.
“this is why if you're instead just autonomously doing whatever an LLM says I mean you're going to get into trouble”
Listen at 58:07
Cal Newporton LLM natural-language software interfacesNegative1:01:11
LLM natural-language software interfaces are currently too error-prone for products.
“it's wrong too much for it to be a product like that.”
Listen at 1:01:11
Cal Newporton small and large LLMsNeutral1:02:12
Small LLMs perform as well as large ones for most applications.
“Small ones work as well as the big ones for most applications.”
Listen at 1:02:12
Cal Newporton LLM application harnessesPositive1:02:15
Application harnesses are often more important than model training.
“The harnesses are often now more important than the actual training of the thing themselves.”
Listen at 1:02:15
Cal Newporton large LLMsNegative1:04:35
Building increasingly large LLMs is economically unsound without human-level intelligence.
“If you're not going to be as smart as a human, building bigger LLMs is a bad business model.”
Listen at 1:04:35
Cal Newporton AI systems for programmingPositive1:05:36
Future AI systems will address unreliability through narrower tuning and more precise specifications.
“I think ultimately to solve this unreliability problem and this over-specking problem”
Listen at 1:05:36
Cal Newporton future AI advancesPositive1:06:51
Many future AI advances will use non-LLM architectures.
“a lot of the more impressive stuff that's going to happen in AI won't be LLM driven”
Listen at 1:06:51
Cal Newporton trained AI modelsNeutral1:08:12
A trained model remains unchanged until it is retrained.
“a model doesn't change. Once it's trained, it's there until you retrain it.”
Listen at 1:08:12
Cal Newporton LLMNeutral1:09:03
Prompts do not update an LLM’s learned parameters or world model.
“Your prompts do not change the LLM. It does not learn. It does not update priors. It does not create a new model of the world.”
Listen at 1:09:03
Cal Newporton LLMs and AI industry investmentNegative1:10:20
Treating LLMs as synonymous with AI resembles investing entirely in AOL as the internet emerged.
“this idea that we stuck to the first thing. And I think it would be the equivalent of being in 1994 and being like, man, this internet thing looks really, I didn't really think about it until I encountered AOL. I'm putting all my chips into AOL”
Listen at 1:10:20

Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.