ffmpeg is mostly known for being the swiss army knife of video processing. In recent years, as the advancement of AI-based video processing starts to become popular, ffmpeg has introduced many of these functionalities through a number of filters. These filters has dramatically lowered the technical barrier for doing AI-based video processing.

In this talk, we’ll examine the various existing filters like SuperResolution or Derain. We will also talk about our experience creating a scene classification filter – training the models, implementing the filter using ffmpeg’s tensorflow backend, and using GPUs to run the model for live streams. This filter is open source, and currently detects content types such as professional soccer games or adult content. We will share benchmarks and our learnings in tuning the performance for this filter.

Presented at Demuxed 2021.