Dwarkesh Podcast
Dwarkesh Podcast

Jun 19, 2026 · 12 min

Sample efficiency defines the true trajectory of artificial intelligence

The data black hole at the center of AI

Understanding whether AI progress is driven by scaling or genuine algorithmic learning efficiency determines the ultimate limits of machine intelligence.

3 key takeaways
  1. 1Sample efficiency serves as a core metric for defining and measuring true algorithmic intelligence.
  2. 2Recent AI progress may rely more on scaling massive data and compute than genuine learning improvements.
  3. 3The reliance on massive data scaling raises critical questions about the future trajectory of AI development.

The brief

Recent breakthroughs in artificial intelligence have stunned the world, but they mask a fundamental question about how these systems actually learn. We must ask whether AI is truly getting smarter or just consuming massive amounts of data.

True intelligence can be defined by sample efficiency, which measures how effectively an algorithm learns from limited data. While humans can master concepts from a few examples, modern AI still requires vast oceans of information to achieve competency.

This raises a critical tension for the future of technology. If progress relies entirely on scaling up data and compute rather than improving algorithmic efficiency, we may soon hit a hard ceiling as high-quality training data runs out.

What was said on this episode

29 statements · 12 positive · 12 negative · 1 mixed · 4 neutral

  1. Dwarkesh Patelon AI training sample efficiencyNegative0:07

    AI training sample efficiency has not clearly improved much in recent years

    “it's actually not clear that we've made that much progress in training sample efficiency over the last few years”

    Listen at 0:07

  2. Dwarkesh Patelon AI progressPositive0:18

    AI progress has mainly come from better data and more data-development compute

    “The main way that AIs have been getting better is from adding more and better data and scaling the compute required to develop that data in the first place.”

    Listen at 0:18

  3. Reinforcement learning requires models to have prior probability for correct solutions.

    “For this process to work, the model must have at least some prior probability to anticipate the correct solution in the first place.”

    Listen at 0:47

  4. Dwarkesh Patelon Human expert training dataNegative0:53

    AI competence across fields requires vast human-expert trajectories for each skill.

    “Which is why you need mind stretching amounts of human expert trajectories in every single field and skill that you want the model to eventually be competent in.”

    Listen at 0:53

  5. Dwarkesh Patelon AI expert training dataNegative0:54

    AI competence across skills requires vast amounts of expert human trajectories

    “you need mind stretching amounts of human expert trajectories in every single field and skill that you want the model to eventually be competent in”

    Listen at 0:54

  6. Dwarkesh Patelon AI skill data productionNegative1:28

    Each AI skill requires at least hundreds of human experts producing examples, rubrics, and reasoning.

    “Each skill corresponds to at least hundreds of human experts who are generating example completions, writing rubrics and explaining their chain of thought.”

    Listen at 1:28

  7. Dwarkesh Patelon AI data-labeling and RL-environment industryPositive1:36

    The AI data and reinforcement-learning-environment industry earns billions annually and may soon reach tens of billions.

    “There's a reason that the data industry that is producing these expert labels and the RL environments in which these meticulously cataloged skills can congeal, is earning billions a year in revenue, soon to be deca. Billions.”

    Listen at 1:36

  8. Dwarkesh Patelon AI training-data industryPositive1:37

    The expert-label and RL-environment data industry earns billions annually

    “the data industry that is producing these expert labels and the RL environments in which these meticulously cataloged skills can congeal, is earning billions a year in revenue”

    Listen at 1:37

  9. Data-driven progress makes it relatively easy for open models and laggards to approach frontier systems.

    “I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress.”

    Listen at 2:29

  10. Dwarkesh Patelon AI progressPositive2:36

    Data is the primary driver of AI progress

    “data is the real driver of progress”

    Listen at 2:36

  11. Dwarkesh Patelon Public AI APIsMixed2:39

    Public APIs make AI training data easier to distill than technical optimization methods.

    “And data can be easily distilled from public APIs, whereas hyperparameters and training tricks and architectural optimizations cannot.”

    Listen at 2:39

  12. Frontier models train on tens to hundreds of trillions of tokens

    “these frontier models are trained on somewhere between tens to hundreds of trillions of tokens”

    Listen at 3:27

  13. Dwarkesh Patelon Current AI systemsNegative3:51

    Current AI systems learn substantially less efficiently than humans.

    “But the reason we can't do this is that our AIs learn much less efficiently than we do.”

    Listen at 3:51

  14. Dwarkesh Patelon Current AI systemsNegative3:52

    Current AI systems learn less efficiently than humans

    “our AIs learn much less efficiently than we do”

    Listen at 3:52

  15. Dwarkesh Patelon AI robotics systemsNegative3:56

    Millions of demonstration hours do not enable AI to perform complex open-ended tasks.

    “And even with the millions of hours of demonstrations that we've collected, this is not enough to to allow them to perform complex open ended tasks.”

    Listen at 3:56

  16. Dwarkesh Patelon Waymo and Tesla self-driving modelsPositive4:13

    Human driving requires far less training data than Waymo and Tesla models use

    “that is still three to four orders of magnitude less data than Waymo and Tesla are using to train their self driving car models”

    Listen at 4:13

  17. Dwarkesh Patelon Human genomeNeutral4:46

    The human genome is approximately three gigabytes, with one to two percent protein-coding.

    “Our genome is only 3 gigabytes big and only 1 to 2% of it is protein coding.”

    Listen at 4:46

  18. Dwarkesh Patelon Human brain developmentNeutral4:59

    Evolution supplied learning hyperparameters and loss functions, while individuals build their brain connectomes.

    “I think the closer analogy is more that evolution found the right hyperparameters and the right loss functions and that within our lifetime we are still from scratch building up the connectome in our brain”

    Listen at 4:59

  19. Evolution supplied humans with effective hyperparameters and loss functions

    “evolution found the right hyperparameters and the right loss functions”

    Listen at 5:02

  20. Dwarkesh Patelon Pretrained AI modelsNegative5:39

    Pretrained AI still requires enormous data for each additional skill.

    “But these AIs, even once they're pre trained, still require enormous amounts of data to learn the next marginal skill and the next marginal skill after that.”

    Listen at 5:39

  21. Dwarkesh Patelon Blind and deaf peoplePositive6:00

    Blind and deaf people retain general intelligence without typical sensory input

    “blind and deaf people who have been cut off from all the sensory information still have general intelligence”

    Listen at 6:00

  22. Dwarkesh Patelon Human-AI data gapNegative6:25

    The human-AI data gap may exceed one millionfold

    “even the million fold difference that we calculated earlier might be an understatement”

    Listen at 6:25

  23. Dwarkesh Patelon AI model scalingNegative7:14

    Unlimited parameter growth would reduce required data by only tenfold at equal loss

    “Even if you increase the number of parameters by infinity, that would only decrease by a factor of 10 the amount of data that you need in order to keep the same loss.”

    Listen at 7:14

  24. Dwarkesh Patelon Human versus AI sample efficiencyNegative7:23

    Humans are thousands to millions of times more sample-efficient than current AI models.

    “Humans are somewhere between thousands to millions of times more sample efficient than these models.”

    Listen at 7:23

  25. Dwarkesh Patelon Scaling current AI modelsNegative7:28

    Scaling current model size alone cannot close the human-AI sample-efficiency gap.

    “So scaling the size of current models simply can't make up for that discrepancy.”

    Listen at 7:28

  26. Dwarkesh Patelon Common white-collar tasks in AI trainingPositive9:16

    Training on common tasks creates substantial economic value

    “it does suggest that there's an enormous amount of value from bringing into distribution these kinds of common tasks”

    Listen at 9:16

  27. Dwarkesh Patelon AI training economicsPositive10:05

    AI training can remain highly inefficient while still being profitable

    “we can be ludicrously inefficient in training them up and still be wildly in the green”

    Listen at 10:05

  28. Dwarkesh Patelon Human software engineersPositive10:44

    Demand for human software engineers will be higher in 2027

    “I would be willing to bet that there's overall more demand for human software engineers in 2027 than there is right now”

    Listen at 10:44

  29. Dwarkesh Patelon AI and human software engineersPositive10:50

    AI complements human software engineers and increases their demand

    “largely due to the complementary input of AI”

    Listen at 10:50

Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.

Weekly email

State of the Pod, every Monday

The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.

Listen to the full episode and explore every guest, topic, and moment on PodLume.