Dwarkesh Podcast
Dwarkesh Podcast

Jun 26, 2026 · 20 min

AI labs bet on reinforcement learning to solve long-horizon tasks

The next big breakthrough will be AIs learning on the job

The core research bet of major AI labs relies on reinforcement learning, but technical bottlenecks in context length and generalization could stall the path to true autonomy.

3 key takeaways
  1. 1AI labs are training agents across millions of verifiable environments to achieve artificial general intelligence.
  2. 2Serving models at longer context lengths than they were trained on leads to significant performance degradation.
  3. 3Short-horizon reinforcement learning training may fail to scale to complex, open-ended real-world endeavors.

The brief

Major AI labs are betting their futures on training agents across millions of verifiable tasks and reinforcement learning environments to unlock artificial general intelligence.

A critical technical bottleneck lies in context length, where training models at a short context length but serving them at a longer one causes severe performance degradation.

The ultimate test for reinforcement learning is generalization, specifically whether training on short-horizon tasks can scale to long-horizon, open-ended real-world challenges.

While AI agents may master specific white-collar tasks, it remains highly uncertain if they can generalize to highly complex, open-ended endeavors like building a business.

What was said on this episode

35 statements · 22 positive · 7 negative · 1 mixed · 5 neutral

  1. AI agents are increasingly solving ambitious problems over longer time spans.

    “AI agents are able to solve more and more ambitious problems over longer and longer time spans.”

    Listen at 1:17

  2. Dwarkesh Patelon In-context learningPositive1:35

    Strong in-context learning could eliminate the need to consolidate job learning into model weights.

    “if in context learning gets so good across longer and longer time horizons, then you don't need to distill back everything the model is learning on the job into the weights.”

    Listen at 1:35

  3. Dwarkesh Patelon AI context windowsPositive2:07

    AI systems may achieve effectively infinite-feeling context windows within several years.

    “with a couple more years of progress, we might have what feels like infinitely large context windows.”

    Listen at 2:07

  4. Dwarkesh Patelon AI training environmentsNeutral3:09

    Verifiability alone is insufficient; domains must also support grindable training.

    “it is not enough for a domain to be verifiable. It's it also has to be very grindable”

    Listen at 3:09

  5. Dwarkesh Patelon AI computer usePositive4:01

    Highly capable coding AIs will accelerate computer-use progress by building faithful application clones.

    “Of course, once AIs get good enough at coding themselves to build these clones with extremely high fidelity, then I'm sure the computer use will make quicker progress than it is right now.”

    Listen at 4:01

  6. Dwarkesh Patelon AI-built application clonesPositive4:02

    AI-built high-fidelity application clones would accelerate computer-use progress.

    “once AIs get good enough at coding themselves to build these clones with extremely high fidelity, then I'm sure the computer use will make quicker progress than it is right now.”

    Listen at 4:02

  7. Dwarkesh Patelon AI computer usePositive4:20

    AI computer use may be solved soon.

    “while computer use itself may soon be solved”

    Listen at 4:20

  8. Dwarkesh Patelon Replayable training targetsNegative4:25

    Models will struggle in domains lacking replayable training targets.

    “unless you can build a very replayable training target for a domain, the models will struggle to make much progress.”

    Listen at 4:25

  9. Dwarkesh Patelon AI sample efficiencyNeutral5:32

    Sparse, idiosyncratic real-world data makes sample efficiency necessary for AI proficiency.

    “because of the idiosyncratic and sparse nature of data in most domains in the world, you need sample efficiency in order to get proficient.”

    Listen at 5:32

  10. Dwarkesh Patelon AI sample efficiencyPositive5:37

    AI systems need sample efficiency to become proficient in real-world domains.

    “you need sample efficiency in order to get proficient.”

    Listen at 5:37

  11. Dwarkesh Patelon RLVR-trained agentsPositive6:13

    Training on reproducible environments will create general agents that execute plans and learn rapidly.

    “if you train on enough containerized, reproducible environments, you will develop a very general agent that can make it execute plans and learn rapidly from new information”

    Listen at 6:13

  12. Dwarkesh Patelon RLVRPositive6:27

    An extensively RLVR-trained AI could outperform LBJ’s political advice in 1948 Texas.

    “If you dropped this endlessly RLVR'd AI into Texas politics in 1948, it could give you better advice than LBJ about winning the Senate seat.”

    Listen at 6:27

  13. Dwarkesh Patelon RLVRPositive6:35

    An RLVR-trained AI could build SpaceX with $100 million starting in 2002.

    “if you gave it a hundred million dollars in 2002 and let it cook, it would build SpaceX for you.”

    Listen at 6:35

  14. Dwarkesh Patelon Long-horizon RLNegative7:18

    Short-horizon RL training may fail to generalize to long-horizon performance.

    “it seems like he's saying that short horizon RL training doesn't necessarily generalize to long horizon RL performance.”

    Listen at 7:18

  15. Dwarkesh Patelon Short-horizon RL trainingNegative7:20

    Short-horizon RL training may not generalize to long-horizon performance.

    “short horizon RL training doesn't necessarily generalize to long horizon RL performance.”

    Listen at 7:20

  16. Dwarkesh Patelon AI inference computeNegative7:50

    Inference consumes 30–50% of lab compute but currently does not improve models.

    “Around 30 to 50% of a lab's compute goes to inference and that compute is currently not playing any productive role in helping improve the model.”

    Listen at 7:50

  17. Dwarkesh Patelon AI deployment dataPositive8:01

    Deployment reveals the most valuable information for improving AI models.

    “it is only in deployment that the most valuable bits of information which your model could learn from are actually revealed.”

    Listen at 8:01

  18. Dwarkesh Patelon AI continual learningNeutral8:41

    Continual learning requires updating model weights.

    “this kind of continual learning requires going back to the weights.”

    Listen at 8:41

  19. Dwarkesh Patelon AI KV cachesNegative8:45

    Continual learning through indefinitely expanding KV caches is not scalable.

    “AIs can't just keep building up a bigger and bigger KV cache as they learn from more and more users. That's just not scalable”

    Listen at 8:45

  20. Dwarkesh Patelon Gradient updatesNegative9:43

    Gradient updates are highly sample inefficient.

    “gradient updates are super sample inefficient”

    Listen at 9:43

  21. Dwarkesh Patelon Online-learning AI modelsNegative9:46

    Deployed online-learning models currently require repeated identical objectives across millions of users.

    “all of the successfully shipped online learning models have had to learn the exact same thing across millions of users.”

    Listen at 9:46

  22. Dwarkesh Patelon cursor tab modelNeutral9:53

    Cursor Tab learns online from the same objective across over 400 million daily requests.

    “the cursor tab model online learns by predicting the same exact objective for over 400 million requests a day.”

    Listen at 9:53

  23. Dwarkesh Patelon AI continual learningNeutral12:37

    Model architecture is probably not the fundamental bottleneck for continual learning.

    “It doesn't seem to me that architecture is fundamentally what is bottlenecking continual learning.”

    Listen at 12:37

  24. Dwarkesh Patelon On-policy self-distillationPositive13:28

    On-policy self-distillation can improve continual learning without requiring an outer-loop verifiable reward.

    “This is better than RLVR for two reasons. One OPSD doesn't require us to have some outer loop verifiable reward.”

    Listen at 13:28

  25. Dwarkesh Patelon On-policy self-distillationPositive13:28

    On-policy self-distillation is better than RLVR for continual learning.

    “This is better than RLVR for two reasons.”

    Listen at 13:28

  26. Dwarkesh Patelon On-policy self-distillationPositive13:49

    OPSD provides denser supervision than naive reinforcement optimization.

    “OPSD provides a much denser supervision signal than naive RO.”

    Listen at 13:49

  27. RL concentrates model updates on information relevant to achieving outcomes.

    “RL is great at concentrating the update to only what is relevant to getting the outcome right.”

    Listen at 14:32

  28. RL learns less information per sample than supervised learning, potentially benefiting continual learning.

    “RL learns much less information per sample than supervised learning. But this may be a good thing rather than a bad thing.”

    Listen at 14:53

  29. Dwarkesh Patelon AI reality simulationPositive15:36

    Reality simulations could let AIs obtain vastly more simulated training samples per unit time.

    “If the AI can build a good simulation of reality against which to rehearse new skills, or try alternative strategies and reinforce what actually works, then AIs could experience all orders of magnitude more simulated samples in the same wall clock time.”

    Listen at 15:36

  30. Dwarkesh Patelon AI dreamingPositive16:43

    Successful AI dreaming would become a fourth scaling axis.

    “If it works, it would become a fourth axis of scaling alongside pre training, RL and inference time compute”

    Listen at 16:43

  31. Dwarkesh Patelon RLVRPositive17:30

    RLVR training produces agents able to navigate unfamiliar problems and iterate on strategies.

    “All of this RLVR training is producing an agent that can get its bearings when it's thrown at an unfamiliar problem, and it can try different strategies and it can iterate when it hits a roadblock.”

    Listen at 17:30

  32. Dwarkesh Patelon AI continual learningPositive18:38

    Continual learning can expand AI capabilities beyond originally trained verifiable domains.

    “the gamut of AI skills and knowledge and capabilities can expand far beyond the verifiable domains that the model was originally trained against before it was deployed.”

    Listen at 18:38

  33. Dwarkesh Patelon Deployed AI systemsPositive19:08

    Mature AI systems will improve mainly through experience accumulated after deployment.

    “By this point, the main way that AIs get better is not from the training they have received before they are released to the public. Rather, it's from all this experience that they'll be accumulating from being broadly deployed in the economy”

    Listen at 19:08

  34. Dwarkesh Patelon Broadly deployed AI systemsPositive19:09

    Broadly deployed AIs will improve mainly through post-release experience rather than pre-release training.

    “the main way that AIs get better is not from the training they have received before they are released to the public.”

    Listen at 19:09

  35. Dwarkesh Patelon AI systemsPositive19:22

    AI systems will become smarter through interactions with users worldwide.

    “Every time that you interact with an AI, it'll be smarter, not only because it's been learning from your previous sessions, but also because it's been learning from all its interactions with all the other users in the world.”

    Listen at 19:22

Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.

Books & mentions

Dwarkesh Podcast

Deep-dive interviews on the technical frontiers of artificial intelligence and policy

Weekly email

State of the Pod, every Monday

The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.

Listen to the full episode and explore every guest, topic, and moment on PodLume.