In the era of high-definition content, maintaining video quality while minimizing bandwidth consumption is a critical challenge for live streaming platforms. On-device video super resolution (VSR) offers a promising solution to achieve higher resolutions at dramatically reduced bitrates. Traditional upscaling methods, such as bicubic or Lanczos interpolation, often fall short in terms of visual quality. While state-of-the-art machine learning (ML) models can produce incredibly accurate upscaled videos, they typically struggle with real-time performance requirements.

Here, we present a novel ML-based approach for high-quality, on-device VSR, capable of running in real-time on modern hardware. We introduce the architectural details of our WebGPU-powered VSR pipeline, including efficient video frame transfer to the GPU via WebCodecs, optimized shader code, and seamless integration with web-based video players. Additionally, we will showcase the results from live user testing at scale, demonstrating the impact of our solution on user experience and bandwidth savings.

By attending this talk, developers and researchers will get a taste of the latest advancements in on-device VSR and the practical implementation considerations for deploying such solutions in live streaming scenarios. We will explore the challenges and trade-offs involved in designing real-time VSR systems, and discuss future directions for further improvements in quality and performance.

This talk was presented at Demuxed 2024, a conference by and for engineers working in video. Every year we host a conference with lots of great new talks like this in San Francisco. Learn more at https://demuxed.com