Towards Modality Transferable Visual Information Representation with Optimal Model Compression

Abstract

Compactly representing the visual signals is of fundamental importance in various image/video-centered applications. Although numerous approaches were developed for improving the image and video coding performance by removing the redundancies within visual signals, much less work has been dedicated to the transformation of the visual signals to another well-established modality for better representation capability. In this paper, we propose a new scheme for visual signal representation that leverages the philosophy of transferable modality. In particular, the deep learning model, which characterizes and absorbs the statistics of the input scene with online training, could be efficiently represented in the sense of rate-utility optimization to serve as the enhancement layer in the bitstream. As such, the overall performance can be further guaranteed by optimizing the new modality incorporated. The proposed framework is implemented on the state-of-the-art video coding standard (i.e., versatile video coding), and significantly better representation capability has been observed based on extensive evaluations.

arXiv commentAccepted in ACM Multimedia 2020

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Do Deep Nets Really Need to be Deep?
• End-to-end Optimized Image Compression
• Compressing Deep Convolutional Networks using Vector Quantization
• Binarized Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Bayesian Compression for Deep Learning
• Convolutional Neural Network-Based Block Up-sampling for Intra Frame Coding
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Variable Rate Image Compression with Recurrent Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Scalable Model Compression by Entropy Penalized Reparameterization
• Neural Network Compression using Transform Coding and Clustering