Real-world Video Adaptation with Reinforcement Learning

Abstract

Client-side video players employ adaptive bitrate (ABR) algorithms to optimize user quality of experience (QoE). We evaluate recently proposed RL-based ABR methods in Facebook's web-based video streaming platform. Real-world ABR contains several challenges that requires customized designs beyond off-the-shelf RL algorithms -- we implement a scalable neural network architecture that supports videos with arbitrary bitrate encodings; we design a training method to cope with the variance resulting from the stochasticity in network conditions; and we leverage constrained Bayesian optimization for reward shaping in order to optimize the conflicting QoE objectives. In a week-long worldwide deployment with more than 30 million video streaming sessions, our RL approach outperforms the existing human-engineered ABR algorithms.

Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Machine Learning, Long Beach, California, USA, 2019

References in corpus (4)

• A Tutorial on Bayesian Optimization
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Variance Reduction for Reinforcement Learning in Input-Driven Environments
• Towards Characterizing Divergence in Deep Q-Learning

Cited by in corpus (7)

• Interpreting Deep Learning-Based Networking Systems
• Learning Scheduling Algorithms for Data Processing Clusters
• Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement
• Learning in situ: a randomized experiment in video streaming
• Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution
• Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
• Unbiased Experiments in Congested Networks