Deep Learning-Based Video Coding: A Review and A Case Study

Abstract

The past decade has witnessed great success of deep learning technology in many disciplines, especially in computer vision and image processing. However, deep learning-based video coding remains in its infancy. This paper reviews the representative works about using deep learning for image/video coding, which has been an actively developing research area since the year of 2015. We divide the related works into two categories: new coding schemes that are built primarily upon deep networks (deep schemes), and deep network-based coding tools (deep tools) that shall be used within traditional coding schemes or together with traditional coding tools. For deep schemes, pixel probability modeling and auto-encoder are the two approaches, that can be viewed as predictive coding scheme and transform coding scheme, respectively. For deep tools, there have been several proposed techniques using deep learning to perform intra-picture prediction, inter-picture prediction, cross-channel prediction, probability distribution prediction, transform, post- or in-loop filtering, down- and up-sampling, as well as encoding optimizations. In the hope of advocating the research of deep learning-based video coding, we present a case study of our developed prototype video codec, namely Deep Learning Video Coding (DLVC). DLVC features two deep tools that are both based on convolutional neural network (CNN), namely CNN-based in-loop filter (CNN-ILF) and CNN-based block adaptive resolution coding (CNN-BARC). Both tools help improve the compression efficiency by a significant margin. With the two deep tools as well as other non-deep coding tools, DLVC is able to achieve on average 39.6\% and 33.0\% bits saving than HEVC, under random-access and low-delay configurations, respectively. The source code of DLVC has been released for future researches.

References in corpus (32)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• Variational image compression with a scale hyperprior
• End-to-end Optimized Image Compression
• The Perception-Distortion Tradeoff
• Image and Video Compression with Neural Networks: A Review
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• Joint Autoregressive and Hierarchical Priors for Learned Image Compression
• A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding
• Reducing Complexity of HEVC: A Deep Learning Approach
• Context-adaptive Entropy Model for End-to-end Optimized Image Compression
• Lossy Image Compression with Compressive Autoencoders
• Enhancing Quality for HEVC Compressed Videos
• Multi-Frame Quality Enhancement for Compressed Video
• Convolutional Neural Network-Based Block Up-sampling for Intra Frame Coding
• Learning for Video Compression
• Variable Rate Image Compression with Recurrent Neural Networks
• Towards Image Understanding from Deep Compression without Decoding
• RNADE: The real-valued neural autoregressive density-estimator
• Generative Image Modeling Using Spatial LSTMs
• A Deep and Tractable Density Estimator
• Neural network-based arithmetic coding of intra prediction modes in HEVC
• A Convolutional Neural Network Approach for Half-Pel Interpolation in Video Coding
• Learning based Facial Image Compression with Semantic Fidelity Metric
• On The Classification-Distortion-Perception Tradeoff
• Target-Quality Image Compression with Recurrent, Convolutional Neural Networks
• Learning Representations by Maximizing Compression
• Adversarial Video Compression Guided by Soft Edge Detection
• PixelCNN Models with Auxiliary Variables for Natural Image Modeling
• Enlarging Context with Low Cost: Efficient Arithmetic Coding with Trimmed Convolution

Cited by in corpus (9)

• BVI-DVC: A Training Database for Deep Video Compression
• MFRNet: A New CNN Architecture for Post-Processing and In-loop Filtering
• Video Compression with CNN-based Post Processing
• ViSTRA2: Video Coding using Spatial Resolution and Effective Bit Depth Adaptation
• Recent Advances on HEVC Inter-frame Coding: From Optimization to Implementation and Beyond
• CVEGAN: A Perceptually-inspired GAN for Compressed Video Enhancement
• Deep Video Codec Control for Vision Models
• A motion-based compression algorithm for resource-constrained video camera traps
• GenStream: Semantic Streaming Framework for Generative Reconstruction of Human-centric Media