← LLM inference batch size
What podcasts say about LLM inference batch size
Every statement, with the speaker, the exact quote and the moment it was said.
What experts have said about LLM inference batch size
3 statements · 3 neutral
The batch size should exceed roughly 300 times the model sparsity ratio.
“batch size needs to be bigger than approximately 300 times sparsity”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 19:17
Practical batch sizes should be roughly two to three times the theoretical balance point.
“take this and maybe double it or triple it”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 19:51
The balance-point batch size depends on sparsity rather than overall model scale.
“beyond that it only depends on sparsity, not on scale”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 26:04
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.