Boosting Neural Image Compression for Machines Using Latent Space Masking
arXiv:2112.08168 · doi:10.1109/TCSVT.2022.3195322
Abstract
Today, many image coding scenarios do not have a human as final intended user, but rather a machine fulfilling computer vision tasks on the decoded image. Thereby, the primary goal is not to keep visual quality but maintain the task accuracy of the machine for a given bitrate. Due to the tremendous progress of deep neural networks setting benchmarking results, mostly neural networks are employed to solve the analysis tasks at the decoder side. Moreover, neural networks have also found their way into the field of image compression recently. These two developments allow for an end-to-end training of the neural compression network for an analysis network as information sink. Therefore, we first roll out such a training with a task-specific loss to enhance the coding performance of neural compression networks. Compared to the standard VVC, 41.4% of bitrate are saved by this method for Mask R-CNN as analysis network on the uncompressed Cityscapes dataset. As a main contribution, we propose LSMnet, a network that runs in parallel to the encoder network and masks out elements of the latent space that are presumably not required for the analysis network. By this approach, additional 27.3% of bitrate are saved compared to the basic neural compression network optimized with the task loss. In addition, we are the first to utilize a feature-based distortion in the training loss within the context of machine-to-machine communication, which allows for a training without annotated data. We provide extensive analyses on the Cityscapes dataset including cross-evaluation with different analysis networks and present exemplary visual results. Inference code and pre-trained models are published at https://github.com/FAU-LMS/NCN_for_M2M.
12 pages, 9 figures, 3 tables; This work has been accepted for IEEE T-CSVT special issue "Learned Visual Data Compression for both Human and Machine"
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Causal Contextual Prediction for Learned Image Compression
- Image coding for machines: an end-to-end learned approach
- Video Coding for Machines with Feature-Based Rate-Distortion Optimization
- Saliency-Driven Versatile Video Coding for Neural Object Detection
- On Intra Video Coding and In-loop Filtering for Neural Object Detection Networks
- Robust Deep Neural Object Detection and Segmentation for Automotive Driving Scenario with Compressed Image Data
- Analysis of Neural Image Compression Networks for Machine-to-Machine Communication