Dwarkesh Podcast
Dwarkesh Podcast

May 15, 2026 · 2h 38m

AlphaGo architecture reveals how reinforcement learning drives artificial general intelligence

Eric Jang – Building AlphaGo from scratch

Understanding the architectural leap from board game AI to self-improving LLMs reveals how modern systems are beginning to automate their own scientific discovery.

3 key takeaways
  1. 1Monte Carlo Tree Search and self-play provide the foundational primitives necessary for scaling machine intelligence beyond human limits.
  2. 2Inference-time scaling and reinforcement learning are shifting the paradigm of how modern large language models are trained and deployed.
  3. 3Automated LLM loops successfully manage complex hyperparameter optimization but still lack high-level human scientific intuition.

Don't miss

Eric Jang explains how automated LLM loops conduct AI research and where they hit a wall in scientific intuition.

The brief

AI researcher Eric Jang joins Dwarkesh Patel to break down the mechanics of AlphaGo, demonstrating how Monte Carlo Tree Search and self-play serve as fundamental primitives for building intelligence.

The transition from board game AI to modern large language models shows how reinforcement learning and inference-time scaling are actively reshaping the path toward artificial general intelligence.

Jang details his hands-on experience using automated LLM loops to conduct AI research, showcasing how automated systems can already handle complex tasks like hyperparameter optimization.

While automated loops excel at optimization, the frontier of AI development still faces major bottlenecks in replicating high-level human scientific intuition and breakthrough reasoning.

What was said on this episode

72 statements · 50 positive · 14 negative · 8 neutral

  1. Eric Zhangon KataGoPositive1:49

    KataGo reduced training compute for strong Go bots by approximately forty times.

    “achieved a 40x reduction in compute needed to train a really strong GoBot”

    Listen at 1:49

  2. Eric Zhangon LLM coding for AlphaGo reproductionPositive2:04

    LLM coding reduces AlphaGo-like reproduction costs from millions to thousands of dollars.

    “thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented computer.”

    Listen at 2:04

  3. Eric Zhangon LLM coding for Go AIPositive2:05

    LLM coding has reduced AlphaGo-like implementation costs from millions to thousands of dollars.

    “what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented computer”

    Listen at 2:05

  4. Eric Zhangon Go exhaustive searchNegative11:57

    Go was long considered computationally intractable for exhaustive search.

    “computer scientists for many years thought that Go was not a tractable problem.”

    Listen at 11:57

  5. Eric Zhangon Human Go playersPositive25:21

    Human Go players use an implicit value function to evaluate whether a board position is winnable.

    “humans as implicitly having a neural network called a value function that basically takes in a board state and then it kind of evaluates key win”

    Listen at 25:21

  6. Eric Zhangon Go value functionPositive26:30

    A trained value function can resolve Go positions without exhaustively searching deeply.

    “you can train a value function to look at a board and quickly resolve the game without playing out all of these trees into a very deep search depth”

    Listen at 26:30

  7. Eric Zhangon AlphaGoPositive27:47

    AlphaGo makes both Go tree breadth and search depth computationally tractable.

    “AlphaGo gives us a way to basically shrink both of those to be very tractable”

    Listen at 27:47

  8. Eric Zhangon ResNets versus transformers for GoPositive33:14

    Residual networks outperform transformers for low-budget Go experiments.

    “my experience is that resnets still kind of outperform transformers and kind of give you more bang for the buck at lower budgets”

    Listen at 33:14

  9. Eric Zhangon ResNets versus Transformers for GoPositive33:14

    ResNets outperform Transformers for low-budget Go models.

    “My experience is that resnets still kind of outperform transformers and kind of give you more bang for the buck at lower budgets.”

    Listen at 33:14

  10. Eric Zhangon Transformers versus residual convolutional networksPositive33:27

    Transformers outperform residual convolutional networks when global context is needed.

    “generally transformers start to outperform residual convolutional networks when you want more global context.”

    Listen at 33:27

  11. Eric Zhangon Transformers versus residual convolutional networksPositive33:28

    Transformers outperform residual convolutional networks when tasks require more global context.

    “transformers start to outperform residual convolutional networks when you want more global context”

    Listen at 33:28

  12. Eric Zhangon Transformers in computer visionNegative34:40

    Transformers require more data to learn invariant local features in vision tasks.

    “you do need more data there so that you can kind of learn through data the sort of invariant, local, local features”

    Listen at 34:40

  13. Eric Zhangon Nash equilibrium strategies in perfect-information gamesPositive36:06

    Perfect-information games have Nash strategies no worse than alternatives.

    “in perfect information games there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.”

    Listen at 36:06

  14. Eric Zhangon Perfect-information gamesPositive36:06

    Perfect-information games have Nash-equilibrium strategies no worse than other strategies.

    “in perfect information games there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy”

    Listen at 36:06

  15. Eric Zhangon Go Nash-equilibrium strategyPositive36:43

    The Nash-equilibrium strategy used by Go agents appears unbeatable by human strategies.

    “The Nash equilibrium seems to be superhuman. No human strategy seems to be able to beat it.”

    Listen at 36:43

  16. Eric Zhangon Imperfect-information gamesNeutral37:19

    Imperfect-information games require historical context to model behavior.

    “in games of imperfect information or partial observability, then you do need some context to build a model.”

    Listen at 37:19

  17. Eric Zhangon Human-data-trained Go modelPositive42:07

    A model trained on human Go data is already a strong Go player.

    “this model is already a pretty good go player.”

    Listen at 42:07

  18. Eric Zhangon Expert-trained Go policy networkPositive42:11

    A policy network trained on expert games can play Go quickly and strongly without search.

    “if you just take this policy recommendation and take the Argmax over, if this is the probabilities, if you take the Argmax and you just take this action as your go play, it'll be a very, very fast go player that doesn't think in terms of reasoning steps. It just kind of shoots from the hip and it'll be a very strong go player, which is already quite miraculous”

    Listen at 42:11

  19. Eric Zhangon Raw Go policy modelPositive42:25

    A raw policy model can play Go quickly without explicit search.

    “it'll be a very, very fast go player that doesn't think in terms of reasoning steps.”

    Listen at 42:25

  20. Eric Zhangon Modern Go botsPositive44:29

    Modern Go bots require relatively little test-time compute.

    “modern go bots don't need that much compute at test time.”

    Listen at 44:29

  21. Eric Zhangon Modern Go botsPositive44:29

    Modern Go bots require relatively little test-time compute.

    “modern go bots don't need that much compute at test time”

    Listen at 44:29

  22. Eric Zhangon Explicit policy head in MCTSPositive57:31

    Explicit policy modeling improves Monte Carlo tree search feedback and recursive self-improvement.

    “having this as an explicit entity you're modeling rather than an implicit normalization over your value, is a good idea”

    Listen at 57:31

  23. Eric Zhangon AlphaGo policy distillationPositive1:05:35

    AlphaGo training distills the outcome of search into the neural network policy.

    “just train this to approximate the outcome of 1000 steps of search”

    Listen at 1:05:35

  24. Eric Zhangon MCTS policy improvementPositive1:07:14

    Training on MCTS-improved actions improves the policy.

    “it does guarantee that if you take these tuples as training data so that you retrain your policy network to predict these ones instead of these ones, you're going to do better.”

    Listen at 1:07:14

  25. Eric Zhangon Monte Carlo tree searchNeutral1:09:36

    MCTS convergence is guaranteed only with infinitely many simulations.

    “It's only guaranteed to converge when you kind of take N to infinity.”

    Listen at 1:09:36

  26. Eric Zhangon Monte Carlo tree searchNeutral1:09:36

    MCTS convergence is guaranteed only in the limit of infinitely many simulations.

    “It's only guaranteed to converge when you kind of take N to infinity.”

    Listen at 1:09:36

  27. Eric Zhangon Monte Carlo tree searchNegative1:09:50

    MCTS does not guarantee policy improvement under inaccurate estimates.

    “And so that's why it's not a guarantee to improve.”

    Listen at 1:09:50

  28. Eric Zhangon Monte Carlo tree searchNegative1:09:51

    Monte Carlo tree search does not always improve the policy network.

    “it's not a guarantee to improve”

    Listen at 1:09:51

  29. Eric Zhangon AlphaGo-style self-play trainingPositive1:10:04

    Resolving 10% of games fully can supply late-game training data.

    “for 10% of the games you prevent the bots from resigning and you just say resolve it to the end”

    Listen at 1:10:04

  30. Eric Zhangon Go value function initializationPositive1:11:42

    Practitioners should first establish a strong value function before investing heavily in MCTS.

    “You want to first make sure that this is good before you invest a lot of cycles doing mcts”

    Listen at 1:11:42

  31. Eric Zhangon MCTS with inaccurate value predictionsNegative1:11:47

    Search should not be applied before value predictions are reliable.

    “It doesn't really make a lot of sense to do search on garbage value predictions.”

    Listen at 1:11:47

  32. Eric Zhangon Random 9x9 Go gamesPositive1:13:46

    Fifty thousand random 9x9 games can train a useful value function.

    “if you play like 50,000 games, you'll actually learn a pretty good value function as well.”

    Listen at 1:13:46

  33. Eric Zhangon Random play on 9x9 Go boardsPositive1:13:46

    Fifty thousand random games on a 9x9 board can train a reasonably good value function.

    “if you play like 50,000 games, you'll actually learn a pretty good value function as well”

    Listen at 1:13:46

  34. Eric Zhangon Monte Carlo tree searchNegative1:14:54

    MCTS fails without a grounded value function.

    “MCTS will fall apart if you don't have a grounding function for, for the value.”

    Listen at 1:14:54

  35. Eric Zhangon Ten-layer neural networkNeutral1:17:34

    A ten-layer network performs only ten sequential reasoning steps.

    “a 10 layer neural network can only do 10 sequential steps of thinking”

    Listen at 1:17:34

  36. Eric Zhangon neural networksPositive1:17:39

    Small neural networks can approximate nearly intractable search through amortization.

    “10 steps of neural network paralyzed distributed representation thinking is able to amortize and approximate to a very, very high fidelity, a nearly intractable search problem.”

    Listen at 1:17:39

  37. Eric Zhangon Approximate solutions to NP-hard problemsPositive1:19:29

    Approximate solutions to hard problems can be extremely useful in practice.

    “in practice it is extremely useful.”

    Listen at 1:19:29

  38. Eric Zhangon Go outcome predictionPositive1:21:30

    Go models can predict the winner despite chaotic exact trajectories.

    “somehow we can predict who's going to win.”

    Listen at 1:21:30

  39. Eric Zhangon Go outcome predictionPositive1:21:30

    Go models can predict the winner despite being unable to predict the exact future board.

    “somehow we can predict who's going to win. And this captures a lot of possibilities here.”

    Listen at 1:21:30

  40. Eric Zhangon Tree search for LLM reasoningNeutral1:47:16

    It remains unresolved whether tree structures can improve LLM reasoning.

    “the jury's still out as to whether this can ever work”

    Listen at 1:47:16

  41. Eric Zhangon PUCT for LLM reasoningNegative1:47:57

    PUCT may be inadequate for guiding LLM reasoning search.

    “for something like LLM reasoning, Pucked might actually not be a good enough heuristic.”

    Listen at 1:47:57

  42. Eric Zhangon Forward search for AI reasoningPositive1:48:43

    Forward search and simulation may return as methods for improving AI reasoning.

    “the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback”

    Listen at 1:48:43

  43. Eric Zhangon Forward search for LLM reasoningPositive1:48:43

    Forward search and simulation may return in LLM reasoning.

    “the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback”

    Listen at 1:48:43

  44. Eric Zhangon Go-style search heuristicsNegative1:50:27

    High-dimensional control and language problems are less suited to Go-style discrete search heuristics.

    “most problems in much higher dimensional action spaces, or something that's combinatorially much bigger, like language, they don't seem as amenable to the kind of discrete action selection heuristics as well as kind of game evaluation type stuff that GO does”

    Listen at 1:50:27

  45. Eric Zhangon Compute-efficient Go bot scalingNegative1:53:29

    Scaling laws have not yet produced Eric’s desired compute-efficient Go bot.

    “I have not been successful so far”

    Listen at 1:53:29

  46. Eric Zhangon Scaling lawsPositive1:53:33

    Scaling laws are most useful when the training recipe and dataset already work.

    “usually when you want scaling loss to work, you want to be in the regime where the recipe already works and the data sets are good”

    Listen at 1:53:33

  47. Eric Zhangon Go bot scaling-law analysisPositive1:54:02

    Scaling-law analysis requires a working strong Go bot first.

    “you actually have to make a strong GoBot first and then use the scaling laws to kind of extrapolate a bit farther into the future.”

    Listen at 1:54:02

  48. Eric Zhangon AI capability developmentNegative1:56:23

    Creating a capability first generally requires more compute than reproducing it later.

    “the compute required to be the first to do something is always much larger than the compute it takes to catch up”

    Listen at 1:56:23

  49. Eric Zhangon AI breakthrough reproductionNeutral1:56:23

    Pioneering AI breakthroughs require more compute than later replication.

    “the compute required to be the first to do something is always much larger than the compute it takes to catch up.”

    Listen at 1:56:23

  50. Eric Zhangon Go bot architecturesNeutral1:59:16

    Architecture choices have limited impact on current strong Go bot performance.

    “architecture choices don't matter that much”

    Listen at 1:59:16

  51. Eric Zhangon Blackwell GPUs for Go trainingPositive1:59:45

    Desktop Blackwell GPUs can train Go bots using roughly half the GPU count of KataGo’s V100 setup.

    “Nvidia GPUs have indeed got faster. So whereas Katago was trained on V1 hundreds, you can train on half the number of desktop Blackwell GPUs and it still works.”

    Listen at 1:59:45

  52. Eric Zhangon Blackwell GPUs for Go trainingPositive1:59:49

    Modern Blackwell GPUs can train comparable Go systems with half as many GPUs.

    “you can train on half the number of desktop Blackwell GPUs and it still works.”

    Listen at 1:59:49

  53. Eric Zhangon 9x9 Go pretrainingPositive2:00:24

    Training on 9x9 boards improves endgame value-function learning.

    “training on 9x9 boards was very nice for resolving endgame value functions.”

    Listen at 2:00:24

  54. Eric Zhangon Replay buffers in reinforcement learningPositive2:04:31

    Replay buffers should contain on-policy states plus off-policy recovery states.

    “your replay buffer really should have the states that your policy would visit, plus some distribution of states that you might drift to and then how to return back to your optimal states”

    Listen at 2:04:31

  55. Eric Zhangon Offline MCTS relabelingPositive2:05:55

    Offline MCTS relabeling of sampled board states can work in practice.

    “in practice, this actually does work.”

    Listen at 2:05:55

  56. Eric Zhangon Off-policy reinforcement learningNegative2:10:51

    Training on unreachable off-policy states wastes model capacity.

    “if the current model is looking at states that it would never reach, then it's kind of wasting capacity”

    Listen at 2:10:51

  57. Eric Zhangon Off-policy training on unreachable statesNegative2:10:51

    Training on unreachable states wastes model capacity.

    “if the current model is looking at states that it would never reach, then it's kind of wasting capacity.”

    Listen at 2:10:51

  58. Eric Zhangon Soft labelsPositive2:18:55

    Soft labels contain more information than one-hot labels.

    “if you have access to the soft targets, the entropy of this distribution is far, far higher than the one hot”

    Listen at 2:18:55

  59. Eric Zhangon Soft targetsPositive2:18:58

    Soft targets contain more information than one-hot labels.

    “the entropy of this distribution is far, far higher than the one hot.”

    Listen at 2:18:58

  60. Eric Zhangon AlphaGoPositive2:20:32

    AlphaGo avoids starting reinforcement learning from zero success and solves exploration through improved labels.

    “you never have to initialize at a 0% success rate and solve the exploration problem of how to get a non zero success rate”

    Listen at 2:20:32

  61. Eric Zhangon AlphaGoPositive2:21:07

    AlphaGo’s supervised learning on improved labels is stable during training.

    “the training is very stable”

    Listen at 2:21:07

  62. Eric Zhangon AI coding modelsPositive2:23:23

    Current AI coding models perform hyperparameter optimization effectively.

    “the models can do a very good job of doing hyperparameter optimization”

    Listen at 2:23:23

  63. Eric Zhangon Frontier modelPositive2:23:23

    Current coding models perform hyperparameter optimization well.

    “the models can do a very good job of doing hyperparameter optimization.”

    Listen at 2:23:23

  64. Eric Zhangon Coding agentsPositive2:24:43

    Coding agents can execute experiments, plot results, and produce reports.

    “it'll go run off and do all the experiments, compile the plot, make a report and suggest what might have caused it or so forth.”

    Listen at 2:24:43

  65. Eric Zhangon Current closed AI modelsNegative2:25:28

    Current closed models are weak at choosing the next experiment.

    “current closed models that we can access, the public can access today, they don't seem to be that great at selecting what the next experiment should be in a given track.”

    Listen at 2:25:28

  66. Eric Zhangon Closed AI research modelsNegative2:25:28

    Current publicly accessible closed models poorly select the next experiment within a research track.

    “current closed models that we can access, the public can access today, they don't seem to be that great at selecting what the next experiment should be in a given track”

    Listen at 2:25:28

  67. Eric Zhangon Mythos-class modelsPositive2:26:03

    More advanced models may substantially improve automated research judgment.

    “with Mythos class models or Mythos models coming online, maybe this just completely changes”

    Listen at 2:26:03

  68. Eric Zhangon Go-trained automated research skillsPositive2:27:02

    Skills learned in Go environments may transfer to bioscience, robotics, and AI research.

    “maybe you can apply them to other domains like biosciences or robotics, or automating AI research.”

    Listen at 2:27:02

  69. Eric Zhangon AI compute scalingPositive2:31:23

    Over the long term, compute is likely the most important determinant of AI performance.

    “in the fullness of time, compute kind of is the single most important determinant on how things work”

    Listen at 2:31:23

  70. Eric Zhangon AI scalingPositive2:31:33

    Scaling energy, compute, and parameters will produce intelligence.

    “as you scale up energy and compute and parameters, intelligence will just fall out of that.”

    Listen at 2:31:33

  71. Eric Zhangon AI compute multipliersNeutral2:32:13

    Individual compute multipliers provide temporary benefits.

    “the sort of benefit of any given compute multiplier is transitory”

    Listen at 2:32:13

  72. Eric Zhangon Game-playing AI researchPositive2:34:22

    Experience developing game-playing AI may transfer positively to building language models.

    “there was some positive transfer from their time working on games and like Atari and Go and Starcraft that now helps them make good lms”

    Listen at 2:34:22

Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.

Weekly email

State of the Pod, every Monday

The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.

Listen to the full episode and explore every guest, topic, and moment on PodLume.