Next-token prediction

Heard in 1 episode across 1 show since Sep 2026

Next-token prediction is a core training objective and generation mechanism in autoregressive language models, where a model is trained to predict the most likely subsequent token (such as a word or subword) given a sequence of preceding tokens. This sequential, probabilistic approach serves as the foundation for modern generative pre-trained transformers (GPTs) and large language models, enabling them to generate coherent, human-like text.