Typical A/B user testing changes client-side behavior between users and compares metrics to evaluate the impact of the change. When applying this principle to video (pipelines/containers/codecs/…) changes, things become more complicated. As the the user selects a version of an asset based on the bucket it is part of, the asset needs to be processed for each bucket and will propagate independently through your CDN. Transient CDN effects, like caching, make that the results in this scenario not reflect the real-life behavior anymore. Suddenly, for a single ingest, the number of cache misses doubles, which negatively impacts the start up time, watch times, favoriting, likes etc. This effect is less outspoken if your CDN cache is not impacting your performance (e.g. you’re Netflix and have high intent and large video buffers). But when you rely on loading video quickly, client-side A/B testing messes with your results. In our Media Centric Experiment approach, assets are bucketed at ingest. With a large number of videos, differences in performance can be attributed to the treatment of both groups. Unfortunately, you can’t just take the ‘average’ of a metric and assume things improve. We’ll discuss the statistical processing (bootstrapping and a Z-test) we apply to confidently make conclusions on the performance of an experiment and make a decision to ship. In this talk, we’ll dive into the issue for A/B testing with video, how we set up our infrastructure to run Media Centric Experiment and explain (in laymen terms) the statistical procedure we follow. Presented at Demuxed 2019 in San Francisco.