Listen to Look: Action Recognition by Previewing Audio

Abstract

In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed video that uses audio as a preview mechanism to eliminate both short-term and long-term visual redundancies. First, we devise an ImgAud2Vid framework that hallucinates clip-level features by distilling from lighter modalities---a single frame and its accompanying audio---reducing short-term temporal redundancy for efficient clip-level recognition. Second, building on ImgAud2Vid, we further propose ImgAud-Skimming, an attention-based long short-term memory network that iteratively selects useful moments in untrimmed videos, reducing long-term temporal redundancy for efficient video-level recognition. Extensive experiments on four action recognition datasets demonstrate that our method achieves the state-of-the-art in terms of both recognition accuracy and speed.

arXiv commentAppears in CVPR 2020; Project page: http://vision.cs.utexas.edu/projects/listen_to_look/

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Recurrent Models of Visual Attention
• Self-Supervised Generation of Spatial Audio for 360 Video
• Look, Listen and Learn
• What Makes Training Multi-Modal Classification Networks Hard?

Cited by in corpus (3)

• Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
• Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
• Can audio-visual integration strengthen robustness under multimodal attacks?