Learning Video Representations using Contrastive Bidirectional Transformer

Abstract

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.

References in corpus (25)

• Representation Learning with Contrastive Predictive Coding
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Learning deep representations by mutual information estimation and maximization
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Exploring the Limits of Language Modeling
• Generating Videos with Scene Dynamics
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Contrastive Multiview Coding
• MINE: Mutual Information Neural Estimation
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Learning Spatiotemporal Features with 3D Convolutional Networks
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
• Video Representation Learning by Dense Predictive Coding
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis

Cited by in corpus (86)

• Transformers in Vision: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• Contrastive Representation Learning: A Framework and Review
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• On Mutual Information Maximization for Representation Learning
• Self-supervised Co-training for Video Representation Learning
• VLP: A Survey on Vision-Language Pre-training
• Hard Negative Mixing for Contrastive Learning
• Attention Bottlenecks for Multimodal Fusion
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Self-Supervised MultiModal Versatile Networks
• UNITER: UNiversal Image-TExt Representation Learning
• TCLR: Temporal Contrastive Learning for Video Representation
• Audiovisual SlowFast Networks for Video Recognition
• A Comprehensive Study of Deep Video Action Recognition
• Spatiotemporal Contrastive Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Contrastive Learning of Structured World Models
• Video Representation Learning with Visual Tempo Consistency
• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• MERLOT: Multimodal Neural Script Knowledge Models
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Support-set bottlenecks for video-text representation learning
• A Multimodal Memes Classification: A Survey and Open Research Issues
• Semi-TCL: Semi-Supervised Track Contrastive Representation Learning
• Cycle-Contrast for Self-Supervised Video Representation Learning
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
• Can Temporal Information Help with Contrastive Self-Supervised Learning?
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Video Understanding as Machine Translation
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Scaling Up Vision-Language Pre-training for Image Captioning
• Learning Video Representations from Textual Web Supervision
• Span-based Localizing Network for Natural Language Video Localization
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
• Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Cross-media Structured Common Space for Multimedia Event Extraction
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
• Environment Predictive Coding for Embodied Agents
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
• Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
• Temporal Query Networks for Fine-grained Video Understanding
• TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
• Tensorized Transformer for Dynamical Systems Modeling
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Learning to Segment Actions from Observation and Narration
• Top1 Solution of QQ Browser 2021 Ai Algorithm Competition Track 1 : Multimodal Video Similarity
• Learning Temporal Dynamics from Cycles in Narrated Video
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• BERT for Large-scale Video Segment Classification with Test-time Augmentation
• Learning to Compare Relation: Semantic Alignment for Few-Shot Learning
• Unsupervised Visual Representation Learning by Tracking Patches in Video
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• SSAN: Separable Self-Attention Network for Video Representation Learning
• Video Contrastive Learning with Global Context
• Self-Supervised Video Representation Learning by Video Incoherence Detection
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
• 15 Keypoints Is All You Need
• Long Short View Feature Decomposition via Contrastive Video Representation Learning
• V-SlowFast Network for Efficient Visual Sound Separation
• Multi-modal Transformer for Video Retrieval
• All You Can Embed: Natural Language based Vehicle Retrieval with Spatio-Temporal Transformers
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping
• A cost-benefit analysis of cross-lingual transfer methods
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• About contrastive unsupervised representation learning for classification and its convergence