Video Compression With Rate-Distortion Autoencoders
arXiv:1908.05717 · doi:10.1109/ICCV.2019.00713
Abstract
In this paper we present a a deep generative model for lossy video compression. We employ a model that consists of a 3D autoencoder with a discrete latent space and an autoregressive prior used for entropy coding. Both autoencoder and prior are trained jointly to minimize a rate-distortion loss, which is closely related to the ELBO used in variational autoencoders. Despite its simplicity, we find that our method outperforms the state-of-the-art learned video compression networks based on motion compensation or interpolation. We systematically evaluate various design choices, such as the use of frame-based or spatio-temporal autoencoders, and the type of autoregressive prior. In addition, we present three extensions of the basic method that demonstrate the benefits over classical approaches to compression. First, we introduce semantic compression, where the model is trained to allocate more bits to objects of interest. Second, we study adaptive compression, where the model is adapted to a domain with limited variability, e.g., videos taken from an autonomous car, to achieve superior compression on that domain. Finally, we introduce multimodal compression, where we demonstrate the effectiveness of our model in joint compression of multiple modalities captured by non-standard imaging sensors, such as quad cameras. We believe that this opens up novel video compression applications, which have not been feasible with classical codecs.
arXiv commentAccepted to ICCV 2019
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Conditional Image Generation with PixelCNN Decoders
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Variational image compression with a scale hyperprior
- End-to-end Optimized Image Compression
- Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
- Lossy Image Compression with Compressive Autoencoders
- Variational Lossy Autoencoder
- Learning for Video Compression
- Fixing a Broken ELBO
- Learning to Inpaint for Image Compression
- Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks
- DVC: An End-to-end Deep Video Compression Framework
Cited by in corpus (42)
- CompressAI: a PyTorch library and evaluation platform for end-to-end compression research
- Temporal Context Mining for Learned Video Compression
- Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression
- Dynamical Variational Autoencoders: A Comprehensive Review
- BVI-DVC: A Training Database for Deep Video Compression
- Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model
- Deep Contextual Video Compression
- MFRNet: A New CNN Architecture for Post-Processing and In-loop Filtering
- A Unified End-to-End Framework for Efficient Deep Image Compression
- Improving Inference for Neural Image Compression
- Advancing Learned Video Compression with In-loop Frame Prediction
- Learning Cross-Scale Weighted Prediction for Efficient Neural Video Compression
- Overfitting for Fun and Profit: Instance-Adaptive Data Compression
- Learned Video Compression via Heterogeneous Deformable Compensation Network
- On Finite Blocklength Lossy Source Coding
- Self-Supervised Variational Auto-Encoders
- CVEGAN: A Perceptually-inspired GAN for Compressed Video Enhancement
- Deep Generative Video Compression
- Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
- Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression
- Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization
- Learning to Compress Videos without Computing Motion
- Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set
- Content Adaptive and Error Propagation Aware Deep Video Compression
- Diffusion Priors In Variational Autoencoders
- Predictive Sampling with Forecasting Autoregressive Models
- Generative Adversarial Networks for Spatio-Spectral Compression of Hyperspectral Images
- Spatiotemporal Entropy Model is All You Need for Learned Video Compression
- Hierarchical Autoregressive Modeling for Neural Video Compression
- Versatile Learned Video Compression
- Improving Deep Video Compression by Resolution-adaptive Flow Coding
- Conditional Entropy Coding for Efficient Video Compression
- Learned Video Compression via Joint Spatial-Temporal Correlation Exploration
- End-to-end Optimized Video Compression with MV-Residual Prediction
- Self-supervised Latent Space Optimization with Nebula Variational Coding
- OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression
- Variational Bayesian Quantization
- Feedback Recurrent AutoEncoder
- Learned transform compression with optimized entropy encoding
- Deep motion estimation for parallel inter-frame prediction in video compression
- Principal Bit Analysis: Autoencoding with Schur-Concave Loss
- Compressing Sensor Data for Remote Assistance of Autonomous Vehicles using Deep Generative Models