Manually testing the functionality of a video player takes time, and when a growing set of features and supported platforms leads to an exploding demand for test executions that cause a bottleneck in your development process, it is time to reconsider.

I will tell the story of how Bitmovin’s test automation team incrementally transformed tedious manual testing into a fully-automated system that executes 150,000 player tests and customer stream conformance tests every day on a worldwide distributed fleet of desktop computers, smartphones, smart TVs, gaming consoles and streaming sticks. More specifically, I will discuss writing a test framework, handling test results and flaky tests, unified cross-platform execution, device selection, software and hardware automation, platform quirks, SDLC integration, remote device usage, getting through lockdowns without interruption, and how we expose the system even to our customers. This talk was presented at Demuxed ’22, a conference for video nerds in San Francisco featuring amazing talks like this one. Demuxed ’22 was made possible by sponsors like our Platinum sponsor Daily (https://daily.co) and organized by people from Mux (https://mux.com). For more information about the conference and community, see https://2022.demuxed.com.