Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise Distillation
arXiv:2312.02605 · doi:10.1109/PCS60826.2024.10566283
Abstract
In recent years, end-to-end learnt video codecs have demonstrated their potential to compete with conventional coding algorithms in term of compression efficiency. However, most learning-based video compression models are associated with high computational complexity and latency, in particular at the decoder side, which limits their deployment in practical applications. In this paper, we present a novel model-agnostic pruning scheme based on gradient decay and adaptive layer-wise distillation. Gradient decay enhances parameter exploration during sparsification whilst preventing runaway sparsity and is superior to the standard Straight-Through Estimation. The adaptive layer-wise distillation regulates the sparse training in various stages based on the distortion of intermediate features. This stage-wise design efficiently updates parameters with minimal computational overhead. The proposed approach has been applied to three popular end-to-end learnt video codecs, FVC, DCVC, and DCVC-HEM. Results confirm that our method yields up to 65% reduction in MACs and 2x speed-up with less than 0.3dB drop in BD-PSNR. Supporting code and supplementary material can be downloaded from: https://jasminepp.github.io/lightweightdvc/
References in corpus (8)
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Video Enhancement with Task-Oriented Flow
- Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression
- Deep Contextual Video Compression
- MFRNet: A New CNN Architecture for Post-Processing and In-loop Filtering
- VCT: A Video Compression Transformer
- EVC: Towards Real-Time Neural Image Compression with Mask Decay
- Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training