Parameter Efficient Multimodal Transformers for Video Representation Learning

Abstract

The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97$\%$, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns together with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips (480 frames) from Kinetics-700 and transfer it to audio-visual classification tasks.

Accepted to ICLR 2021

References in corpus (26)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Gaussian Error Linear Units (GELUs)
• The Kinetics Human Action Video Dataset
• Learning deep representations by mutual information estimation and maximization
• On the Convergence of Adam and Beyond
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Layer Normalization
• Universal Transformers
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Deep Equilibrium Models
• Learning Video Representations using Contrastive Bidirectional Transformer
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• UNITER: UNiversal Image-TExt Representation Learning
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Trellis Networks for Sequence Modeling
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• On Mutual Information in Contrastive Learning for Visual Representations
• Pruning Convolutional Neural Networks with Self-Supervision
• Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events

Cited by in corpus (6)

• Transformers in Vision: A Survey
• Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
• Attention Bottlenecks for Multimodal Fusion
• Video Transformers: A Survey
• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition