HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
arXiv:2005.00200
Abstract
We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.
Accepted by EMNLP 2020
References in corpus (18)
- Decoupled Weight Decay Regularization
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- The Kinetics Human Action Video Dataset
- Learning deep representations by mutual information estimation and maximization
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Exploring the Limits of Language Modeling
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Learning Video Representations using Contrastive Bidirectional Transformer
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- XGPT: Cross-modal Generative Pre-Training for Image Captioning
- Distilling Knowledge Learned in BERT for Text Generation
Cited by in corpus (19)
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
- A Multimodal Memes Classification: A Survey and Open Research Issues
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- Fine-grained Iterative Attention Network for TemporalLanguage Localization in Videos
- Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
- VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
- STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
- Winning the ICCV'2021 VALUE Challenge: Task-aware Ensemble and Transfer Learning with Visual Concepts
- TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
- VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
- Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
- CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
- Video Moment Retrieval with Text Query Considering Many-to-Many Correspondence Using Potentially Relevant Pair
- Survey: Transformer based Video-Language Pre-training
- Listen, Look and Deliberate: Visual context-aware speech recognition using pre-trained text-video representations
- Coarse to Fine: Video Retrieval before Moment Localization
- All You Can Embed: Natural Language based Vehicle Retrieval with Spatio-Temporal Transformers