An HTML5 player for HLS or MPEG-DASH uses the HTML5 video element and the Media Source Extensions API to buffer fragmented MP4 segments and play them out as a smooth video stream. But what actually goes on behind the scenes of these web APIs? What happens when you play or seek the video, and how does the video element maintain smooth playback without dropping frames?

In this talk, I’ll use the WebCodecs and Canvas APIs as low-level building blocks to rebuild parts of the video element and MSE APIs. Using that implementation, I’ll walk through some interesting bits of the playback and buffering logic, and explain what a streaming video player should look out for when using these APIs. Some of the topics that I’ll cover include: How seeking works, how the GOP size affects how fast you can seek, and why you should always put a keyframe at the start of each fMP4 segment. How quality switching works, and why you should try to align your segment boundaries across all qualities. How MSE manages the size of its buffer, and what a player can do to keep this size under control. This talk was presented at Demuxed ’22, a conference for video nerds in San Francisco featuring amazing talks like this one. Demuxed ’22 was made possible by sponsors like our Platinum sponsor Daily (https://daily.co) and organized by people from Mux (https://mux.com). For more information about the conference and community, see https://2022.demuxed.com.