Aug 25, 2026 · 1h 20m
SAIL targets abundant intelligence with cheaper AI inference
Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
The episode examines whether flexible software and overlooked infrastructure can make persistent AI agents economically viable at massive scale.
- 1SAIL prioritizes inference throughput and cost over interactive latency for long-running agents with potentially unlimited demand.
- 2Its scavenger strategy combines adaptable software with fragmented chips, underused capacity, and intermittent power.
- 3Future AI progress may depend less on internet data and more on verifiable tasks, specialized agents, and efficient infrastructure.
Don't miss
Neil Movva argues that demand for intelligence could become effectively unlimited, turning token cost and infrastructure efficiency into the central AI constraint.
The brief
Neil Movva presents SAIL Research as a token factory for long-running agents, arguing that abundant intelligence depends on making inference dramatically cheaper.
The central tradeoff is throughput versus latency: background agents can tolerate delay if batching and parallelism lower the cost of producing enormous volumes of tokens.
SAIL’s software-first approach reaches from GPU kernels and memory to heterogeneous chips, extracting value from hardware that conventional inference providers may overlook.
Movva argues that internet data is nearing saturation, shifting future gains toward agents trained on difficult, verifiable tasks and combinations of specialized intelligences.
The episode’s sharpest bet is that demand for intelligence could become effectively unlimited, making cheap tokens—and the infrastructure behind them—the defining constraint.
What was said on this episode
45 statements · 32 positive · 9 negative · 1 mixed · 3 neutral
SAIL built its LLM stack to maximize GPU efficiency
“we tried to build the entire LLM software stack around peak GPU efficiency”
Listen at 15:16
SAIL aims to produce more tokens per chip than any other company.
“We wanted to squeeze out more tokens from the same chip than anyone else in the world.”
Listen at 15:21
SAIL aims to reach the hardware’s maximum achievable performance
“We're chasing 100% speed of light.”
Listen at 18:21
AI usage will shift from chatbots toward proactive or background agents.
“I believe that's the most profound change we're going to see in the next year. We're going to move away from chatbots to more proactive or background agents.”
Listen at 20:25
AI usage will shift from chatbots toward proactive background agents
“We're going to move away from chatbots to more proactive or background agents.”
Listen at 20:28
Background-agent workloads favor throughput-oriented AI infrastructure
“In that world, it makes a lot more sense to build a stack around throughput.”
Listen at 20:31
Some non-NVIDIA chips offer better compute per dollar
“There are other chips that definitely rank higher than Nvidia on flops per dollar”
Listen at 24:18
NVIDIA is effectively required for tensor parallelism
“Nvidia is basically mandatory for that.”
Listen at 24:31
Cerebras-style wafer-scale systems can reach roughly 1,000 tokens per second
“So there you go. You have a path to a thousand tokens per second.”
Listen at 28:56
Cerebras-style accelerators will be used in hybrid systems
“What happens to them is some hybrid.”
Listen at 29:02
Using the full million-token context window is not generally beneficial.
“I don't think it's actually great to hit the full length.”
Listen at 31:10
KV-cache requirements will constrain high-speed weight-storage architectures.
“However, the KV cache is going to be a big thorn in your side.”
Listen at 31:24
KV-cache requirements will constrain fast inference systems
“the KV cache is going to be a big thorn in your side.”
Listen at 31:24
Hybrid systems may place attention on GPUs and MLPs on Cerebras chips
“You would like to put the attention possibly on the GPU and the MLP on the Cerebris chip.”
Listen at 32:50
Large matrix multiplication on GPUs reaches roughly 70–80% peak utilization and is power-limited.
“That operation runs at 70, 80% of peak utilization, and it's limited not by software, but by power.”
Listen at 42:32
Large matrix multiplications reach roughly 70–80% GPU utilization
“That operation runs at 70, 80% of peak utilization”
Listen at 42:32
Rack-scale computing will drive future AI efficiency and speed
“my belief is that that shape of compute is the future of efficiency and speed.”
Listen at 43:27
SAIL can make any chip viable when priced appropriately
“There's no bad chips. There's only bad pricing. I will make any chip work at the right price.”
Listen at 45:14
Meta and OpenAI have purchased substantial quantities of AMD chips.
“I think AMD is actually somewhat popular amongst some large buyers. I think publicly, Meta and OpenAI have bought a ton of AMD chips.”
Listen at 45:56
AMD AI chips have gained adoption among large buyers
“AMD is actually somewhat popular amongst some large buyers.”
Listen at 45:56
SAIL intends to integrate TPUs and Trainium into heterogeneous serving systems.
“Yes, we love TPUs. We're going to make TPUs work. Yes, we love Tranium. We're going to make Tranium work.”
Listen at 47:20
SAIL intends to integrate TPUs and Trainium into heterogeneous serving
“We're going to make TPUs work. Yes, we love Tranium. We're going to make Tranium work.”
Listen at 47:21
AI inference spending will continue increasing monotonically
“I do think inference spend monotonically increases.”
Listen at 50:45
Training clusters can support inference workloads
“Training is a superset workload over inference.”
Listen at 51:36
Training clusters can support inference, whereas inference clusters may not support training.
“Training is a superset workload over inference. You can make any training cluster work for inference, but maybe not vice versa.”
Listen at 51:36
Inference can run effectively in distributed one-megawatt data centers
“increasingly we're seeing a few new thinkers realize that inference is going to be suitable for these distributed one megawatt data centers.”
Listen at 52:45
SAIL will use distributed small compute pools across the United States for inference.
“We're quite in agreement with that. And we are very happy to buy small pools of compute across the United States and use that as our inference fleet.”
Listen at 52:52
SAIL treats machine performance as improvable rather than fixed.
“We don't accept the state of the machines we work on is fixed.”
Listen at 1:05:08
Neil favors offloading KV cache to flash memory.
“I'm quite passionate about that idea.”
Listen at 1:10:42
Model architectures could be redesigned to enable greater KV-cache flash offloading
“what would we have to change about the model architecture to make offloading KVCache to flash work at a much greater level.”
Listen at 1:10:46
SAIL designs around serving workloads at 1–10 tokens per second
“We have some divergent views on what you can do if you design a system around serving at one to 10 tokens per second, which is our whole North Star.”
Listen at 1:10:54
AI inference cost per token needs to improve by 3–6 orders of magnitude
“We were asking for at least three to six orders of magnitude improvement in cost per token.”
Listen at 1:11:29
Some models can process a trillion tokens for tens of thousands of dollars
“for some size of model, we are approaching a trillion tokens being measured in tens of thousands of dollars.”
Listen at 1:11:38
Global demand for intelligence is effectively unlimited
“There's always demand for intelligence in the world.”
Listen at 1:11:59
Neil is bullish on NVIDIA over the short term
“I am bullish on NVIDIA in the short term.”
Listen at 1:12:27
Investors should not bet against NVIDIA
“you should never bet against them.”
Listen at 1:12:30
Losing access to TSMC would not severely impair Western AI chip supply.
“My contrarian take is that it wouldn't be that bad.”
Listen at 1:13:02
Western chip processes may be at most roughly twice as inefficient as TSMC
“at worst, like maybe 2x, worst performance per watt.”
Listen at 1:13:11
Intel’s process technology is at most roughly twice as inefficient as leading Western processes.
“The gap is just far smaller than you would make it out to be if you follow like the chipboard dialogue.”
Listen at 1:13:15
Modern chip supply is constrained by wafers, HBM, packaging, and power
“There's TSMC wafer capacity, there's HPM capacity, and there's advanced packaging. Maybe a fourth one would be power.”
Listen at 1:14:51
Compute startups should address TSMC, HBM, advanced packaging, and power bottlenecks.
“You should have a great answer to each of those four bottlenecks and how you're going to work around them.”
Listen at 1:14:59
New compute startups should focus on a differentiated niche
“You want to be spiky.”
Listen at 1:15:16
iPhones will ship with less memory and become more expensive.
“I think that iPhones will cut their memory. iPhones are going to go up in price.”
Listen at 1:15:48
iPhone prices will increase because of memory shortages
“iPhones are going to go up in price.”
Listen at 1:15:50
NVIDIA avoids competing directly with its customers
“NVIDIA is really smart about this. They don't compete with their customers.”
Listen at 1:15:57
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.