Audiovisual SlowFast Networks for Video Recognition

Abstract

We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, we perform hierarchical audiovisual synchronization to learn joint audiovisual features. We report state-of-the-art results on six video action classification and detection datasets, perform detailed ablation studies, and show the generalization of AVSlowFast to learn self-supervised audiovisual features. Code will be made available at: https://github.com/facebookresearch/SlowFast.

Technical report

References in corpus (13)

• Improving neural networks by preventing co-adaptation of feature detectors
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Momentum Contrast for Unsupervised Visual Representation Learning
• Spatiotemporal Residual Networks for Video Action Recognition
• Learning Video Representations using Contrastive Bidirectional Transformer
• Video Classification with Channel-Separated Convolutional Networks
• Revisiting Self-Supervised Visual Representation Learning
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• What Makes Training Multi-Modal Classification Networks Hard?
• FBK-HUPBA Submission to the EPIC-Kitchens 2019 Action Recognition Challenge
• Hierarchical Feature Aggregation Networks for Video Action Recognition

Cited by in corpus (27)

• Human Action Recognition from Various Data Modalities: A Review
• Attention Bottlenecks for Multimodal Fusion
• Perceiver: General Perception with Iterative Attention
• A Comprehensive Study of Deep Video Action Recognition
• CLIPort: What and Where Pathways for Robotic Manipulation
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation
• Cycle-Contrast for Self-Supervised Video Representation Learning
• Revisiting Skeleton-based Action Recognition
• Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
• Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
• HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition
• Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• Generic Event Boundary Detection Challenge at CVPR 2021 Technical Report: Cascaded Temporal Attention Network (CASTANET)
• Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition
• Shot Contrastive Self-Supervised Learning for Scene Boundary Detection
• Contrastive Learning of Global-Local Video Representations
• Spatio-Temporal Context for Action Detection
• Can audio-visual integration strengthen robustness under multimodal attacks?
• Where and When: Space-Time Attention for Audio-Visual Explanations
• LSTC: Boosting Atomic Action Detection with Long-Short-Term Context
• PyTorchVideo: A Deep Learning Library for Video Understanding
• Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories
• V-SlowFast Network for Efficient Visual Sound Separation
• The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning