COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

Abstract

Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open-source at https://github.com/gingsi/coot-videotext

arXiv comment27 pages, 5 figures, 19 tables. To be published in the 34th conference on Neural Information Processing Systems (NeurIPS 2020). The first two authors contributed equally to this work

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Gaussian Error Linear Units (GELUs)
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Dual Learning for Machine Translation
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Learnable pooling with Context Gating for video classification
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Towards Automatic Learning of Procedures from Web Instructional Videos
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Unified Vision-Language Pre-Training for Image Captioning and VQA
• Enhancing Sentence Embedding with Generalized Pooling
• No Fuss Distance Metric Learning using Proxies
• Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video
• lamBERT: Language and Action Learning Using Multimodal BERT

Cited by in corpus (8)

• Transformers in Vision: A Survey
• Video Transformers: A Survey
• Robust Domain Misinformation Detection via Multi-modal Feature Alignment
• Hierarchical Matching and Reasoning for Multi-Query Image Retrieval
• Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
• Collaborative Three-Stream Transformers for Video Captioning
• An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video-Text Retrieval
• A Multi-level Alignment Training Scheme for Video-and-Language Grounding