Cross Modal Compression: Towards Human-comprehensible Semantic Compression

Abstract

Traditional image/video compression aims to reduce the transmission/storage cost with signal fidelity as high as possible. However, with the increasing demand for machine analysis and semantic monitoring in recent years, semantic fidelity rather than signal fidelity is becoming another emerging concern in image/video compression. With the recent advances in cross modal translation and generation, in this paper, we propose the cross modal compression~(CMC), a semantic compression framework for visual data, to transform the high redundant visual data~(such as image, video, etc.) into a compact, human-comprehensible domain~(such as text, sketch, semantic map, attributions, etc.), while preserving the semantic. Specifically, we first formulate the CMC problem as a rate-distortion optimization problem. Secondly, we investigate the relationship with the traditional image/video compression and the recent feature compression frameworks, showing the difference between our CMC and these prior frameworks. Then we propose a novel paradigm for CMC to demonstrate its effectiveness. The qualitative and quantitative results show that our proposed CMC can achieve encouraging reconstructed results with an ultrahigh compression ratio, showing better compression performance than the widely used JPEG baseline.

arXiv comment10 pages, 4 figures

References in corpus (7)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Generative Adversarial Text to Image Synthesis
• Image and Video Compression with Neural Networks: A Review
• Learning based Facial Image Compression with Semantic Fidelity Metric
• To Create What You Tell: Generating Videos from Captions