May 6, 2026 · 4h 24m
Open-source developers warn critical video infrastructure is under strain
#496 – FFmpeg: The Incredible Technology Behind Video on the Internet
The global streaming economy rests on open-source software maintained by volunteers, exposing a critical vulnerability in digital infrastructure.
- 1Nearly all digital video streaming relies on open-source projects like FFmpeg and VLC to decode and play media files.
- 2Video compression depends on complex mathematical techniques like the discrete cosine transform to reduce file sizes.
- 3Open-source infrastructure faces growing pressure from big tech demands and security threats without adequate systemic support.
Don't miss
Jean-Baptiste Kempf details the immense pressure open-source developers face from intelligence agencies and tech giants.
The brief
While the world watches the artificial intelligence boom, the quiet infrastructure of the modern internet relies on open-source media engines like FFmpeg and VLC to process virtually all digital video.
Jean-Baptiste Kempf and Kieran Kunhya explain that modern video compression is a delicate balance of math and engineering, relying on complex transforms to shrink massive raw files into streamable data.
The developers warn that maintaining this critical infrastructure is increasingly difficult, as small open-source teams face intense pressure from tech giants and security agencies.
Behind the seamless streaming services we use daily lies a shrinking pool of specialists who still understand the lost art of writing handwritten assembly code for hardware performance.
What was said on this episode
67 statements · 42 positive · 13 negative · 4 mixed · 8 neutral
Up to 45% of video files are not GPU-decodable.
“I think it's up to 45% of files are not GPU decodable.”
Listen at 18:06
Video and audio compression can reduce data by roughly 100 times.
“the compression on video and audio is 100 times”
Listen at 19:45
Reducing color resolution relative to brightness can halve video size without compression.
“we scale down the resolution of the color compared to the brightness. And most of the time and just this, without compression, it divides the size by two.”
Listen at 21:57
Each codec generation achieves about 30% lower size at equivalent quality.
“each generation of the codec is like 30% less for the same quality”
Listen at 22:37
MP4 files generally contain H.264 video and AAC audio.
“MP4 is understood to generally be H264 plus AAC audio 99% of the time.”
Listen at 26:58
VLC and FFmpeg inspect file contents rather than trusting filename formats.
“In VLC and FFmpeg, we discard the file format, right? We look into the file to understand what's in it.”
Listen at 27:33
VLC is engineered to interpret damaged or malformed media files.
“everything in VLC is prepared to work with broken files.”
Listen at 29:13
Video codecs are designed to recover from packet loss and join streams during decoding.
“some of the design goals of a codec is also to be recoverable.”
Listen at 31:06
FFmpeg is the de facto tool for multimedia processing.
“it's really now the de facto tool to process images.”
Listen at 36:14
FFmpeg helped cause the podcast, streaming, and YouTube revolutions by democratizing video technology.
“The podcast and streaming revolution, the YouTube revolution was caused. You know, FFMPEG was a big player in that because it democratized this technology”
Listen at 40:31
Roughly 2,000–3,000 people have contributed to FFmpeg.
“Since the beginning of FFmpeg, probably 2,000 to 3,000 people have contributed from the beginning”
Listen at 43:26
Jean-Baptiste Kempf changed LibVLC’s core license from GPL to LGPL.
“I changed the core of libvlc, which is the engine of vlc, from GPL to lgpl.”
Listen at 49:40
LGPL allows proprietary applications to use the LibVLC library.
“The lgpl, the library GPL as it used to be known, allows you to do that.”
Listen at 50:54
Linux runs approximately 70–80% of servers, including Microsoft Azure servers.
“I'm quite sure 70, 80% of the servers are running Linux.”
Listen at 57:13
Google began using AI to generate security reports about FFmpeg.
“Google started using AI to create security reports on an open source project.”
Listen at 1:16:25
AI-generated security reports can overwhelm FFmpeg volunteers with verbose reports about niche codecs.
“the bug reports are very wordy, they're very, very. It's almost a denial of service by AI generated bug reports on very niche codecs.”
Listen at 1:19:11
FFmpeg donations increased substantially but remain insufficient to fund one full-time developer.
“Donations have increased substantially. They're still not enough to cover even a single full time developer.”
Listen at 1:25:42
Jean-Baptiste Kempf recommends starting open-source work on a subject one loves.
“my answer is always the same. Work on something you love.”
Listen at 1:35:05
FFmpeg may be running on 100 million to one billion CPUs worldwide.
“FFMPEG is probably one of the biggest CPU users in the world. It's probably running as we speak easily order of magnitude 100 million, maybe even a billion CPUs as we speak.”
Listen at 1:37:41
More than 80% of FFmpeg pipelines depend on a VideoLAN project.
“I would put a finger in the air that say 80 plus percent of those pipelines are dependent on a videoland project.”
Listen at 1:42:16
Reverse-engineering one megabyte of binary code can require roughly a month of work.
“one megabyte binary blob to reverse engineer is probably order of magnitude a month of work”
Listen at 1:50:11
FFmpeg uses FATE as its automated testing environment.
“FFmpeg has a system called fate. FFmpeg automated testing Environment.”
Listen at 2:02:28
Handwritten SIMD assembly can make selected functions 10–50 times faster, including one 62-times improvement.
“we have a 10x not percentage 10x to 50x speed improvement. That function is 62x”
Listen at 2:11:25
The David AV1 decoder contains about 30,000 C lines and 240,000 handwritten assembly lines.
“We are talking, talking about 30,000 lines of C, but 240,000 lines of handwritten assembly”
Listen at 2:13:00
About 30% of Netflix video and 50% of YouTube video use AV1.
“30% of the video from Netflix are now in AV1, 50% of YouTube.”
Listen at 2:14:00
Compiler auto-vectorization is multiple times slower than handwritten SIMD assembly.
“it's multiple times slower.”
Listen at 2:15:54
Hardware-specific optimization will remain beyond reliable vibe coding.
“the core thing that you will not be able to vibe code are optimization for the hardware to be as fast as is possible.”
Listen at 2:23:34
Rust cannot match the speedups achieved by handwritten assembly.
“Not assembly by hand? No.”
Listen at 2:30:55
Rust improves on C by enforcing stronger memory-safety and ownership guarantees.
“Rust is extremely good in the sense that it's a better C that cares about memory”
Listen at 2:33:17
Writing code is roughly an order of magnitude easier than reading code.
“It is an order of magnitude easier to write code than read code.”
Listen at 2:34:23
x264 dominates Internet video encoding for the H.264 standard.
“x264 is a video encoder for the H264 video standard. It dominates Internet video”
Listen at 2:56:25
AV1 uses 40–60% less bandwidth than H.264 at equivalent visual quality.
“AV1 saves between 40 and 60% less bandwidth than H264 for the same quality, visual quality”
Listen at 3:06:50
AV1 encoding requires roughly 100 times more CPU cycles than H.264 encoding.
“AV1 encoding is 2 order of magnitude more than H.264 in terms of CPU cycle”
Listen at 3:08:25
YouTube re-encodes popular videos in AV1 to reduce distribution costs.
“the popular video gets re encoded in AV1”
Listen at 3:08:49
VLC is being split into separate sandboxed processes for decoding, demuxing, and filtering.
“what we are trying to do and we're actually doing is splitting VLC into several processes.”
Listen at 3:23:27
Adaptive streaming players lower resolution when network congestion limits download speed.
“locally the player is going to read a lower resolution of it.”
Listen at 3:28:57
Kyber uses one QUIC-over-UDP connection for low-latency multimedia and control streams.
“we take only one Socket one connection, which is a quic protocol based on udp”
Listen at 3:34:42
Kyber aims to enable remote control and observation of machines across distance.
“The goal of Kyber is to make distance disappear because it's either projection of skills or projection of power.”
Listen at 3:37:07
Kyber targets four-millisecond glass-to-glass latency.
“My goal is 4 milliseconds Glass to glass latency.”
Listen at 3:39:48
Kyber achieves seven-millisecond Windows-to-Windows or Windows-to-Mac latency.
“Not so far. We achieve 7 milliseconds from windows to Windows or Windows to Mac.”
Listen at 3:40:19
Millions of robots and drones will operate remotely in the future.
“there is going to be millions of robots or drones are just rolling robots or flying robots or swimming robots”
Listen at 3:41:20
AI models will remotely monitor autonomous vehicles for safety.
“all those remote cars will be tele observed by an AI model”
Listen at 3:41:45
Kyber aims to eliminate distance-related barriers to real-time machine control.
“the goal of Kyber is to make real time control of machine distance disappear”
Listen at 3:42:21
All Kyber components are open source.
“everything on Kyber is open source”
Listen at 3:43:00
Kyber uses a dual commercial and AGPL licensing model.
“It's a dual license, commercial and agpl.”
Listen at 3:43:04
AV2 is intended to reduce bandwidth by 30% versus AV1 at equal quality.
“So AV2 is the next generation of this codec. It's 30% better, right? So if you keep the same quality, you got 30% bandwidth reduction compared to AV1.”
Listen at 3:45:04
Successive major video codec generations improve efficiency by roughly 30%.
“30% each and every generation is the best summary.”
Listen at 3:48:18
AV codecs are mostly free of patent licensing claims.
“So it's mostly true that it's patent free.”
Listen at 3:50:03
Patent licensing can cost YouTube or Netflix hundreds of millions annually.
“for YouTube or Netflix we could talk about hundreds of millions of dollars of licensing for patents per year”
Listen at 3:52:09
High codec patent costs motivated major companies to create their own codecs.
“At 100 million per year I could create my own codec. And this is what they did.”
Listen at 3:52:19
VVC adds patent claims, while AV2 aims to minimize royalties.
“VVC has all the patents of each EVC plus new ones, right. While AV2 tries to be as royalty free as possible.”
Listen at 3:53:24
VLC’s hypothetical patent licensing burden exceeded €200 per user.
“if I had to pay all the licensing fee for VLC I needed to pay more than €200 per user”
Listen at 3:53:53
A new generation of entrepreneurs is emerging in Europe, especially France.
“there is a new generation of entrepreneurs in Europe and mostly in France”
Listen at 3:54:58
Court orders generally do not bankrupt organizations in France or most of Europe.
“answering to a court order does not make you bankrupt”
Listen at 3:57:40
Videoland’s annual lawyer fees are below $10,000.
“the cost of lawyer fees for Vidaland is less than $10,000 per year”
Listen at 3:57:58
VLC would be shut down rather than compromised by external actors.
“if we had to compromise our software, we would shut it down”
Listen at 3:58:22
VLC contains no telemetry.
“There is no telemetry in vlc, right?”
Listen at 4:00:31
C will remain usable in some contexts a thousand years from now.
“It will be like Latin, C will be like Latin. It will be a thing, thing that you learn from the past, but it will still be usable in certain contexts.”
Listen at 4:05:56
The FFV1 codec is lossless and was developed with archival community funding.
“they funded the development of the FFV1 codec. So that's a lossless codex.”
Listen at 4:06:08
FFV1’s losslessness can be mathematically verified.
“They can prove mathematically that it's lossless.”
Listen at 4:10:48
FFmpeg is expected to exist in 100 years; VLC’s survival is uncertain.
“FFmpeg? Yes. VLC maybe.”
Listen at 4:11:10
VLC and FFmpeg will eventually support 3D data.
“at some point they will manage 3D data inside VLT and FFmpeg. Right? It's obvious.”
Listen at 4:13:05
Approximately 150 people contribute annually to VLC and 300 to FFmpeg.
“every year there is around 150 people who contribute to VLC and maybe 300 on FFmpeg”
Listen at 4:15:42
Emerging multimedia technologies initially produce multiple competing standards before hype declines.
“You start, it's a new topic. There is like five different standards. Because everyone starts to do this, the hype goes down.”
Listen at 4:17:21
Earlier 3D audio formats failed, but 3D audio is returning with practical use cases.
“Six or seven years ago, it was everything about 3D. You had the cardboard on Android, you had two audio formats. They're all dead. Right. And now it's coming back with actual use cases”
Listen at 4:17:52
Dolby has shifted from engineering innovation toward legal and licensing activities.
“It used to be an amazing company doing tons of great things with amazing engineers. They defined what sound was and now it's mostly lawyers and licensing things.”
Listen at 4:18:26
The Mars 2020 rover uses FFmpeg to compress pictures.
“FFMPEG is used by the Mars rover, the Mars 2020 rover, to compress pictures.”
Listen at 4:19:35
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.
Books & mentions
Lex Fridman Podcast
Explore more deep-tech conversations with industry pioneers and open-source creators