HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
arXiv:1906.03327
Abstract
Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.
Accepted at ICCV 2019
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
- Learnable pooling with Context Gating for video classification
- Towards Automatic Learning of Procedures from Web Instructional Videos
- How2: A Large-scale Dataset for Multimodal Language Understanding
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
- Learning Language-Visual Embedding for Movie Understanding with Natural-Language
Cited by in corpus (132)
- Learning Transferable Visual Models From Natural Language Supervision
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Human Action Recognition from Various Data Modalities: A Review
- Self-Supervised Representation Learning: Introduction, Advances and Challenges
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- VLP: A Survey on Vision-Language Pre-training
- Learning Video Representations using Contrastive Bidirectional Transformer
- Dual Encoding for Video Retrieval by Text
- Self-Supervised MultiModal Versatile Networks
- ActionCLIP: A New Paradigm for Video Action Recognition
- Video Transformers: A Survey
- Self-Supervised Learning for Videos: A Survey
- MDMMT: Multidomain Multimodal Transformer for Video Retrieval
- Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
- CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
- Spatiotemporal Contrastive Video Representation Learning
- Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
- GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
- SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries
- Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation
- Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
- CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval
- Neural ranking models for document retrieval
- Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
- VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
- Learning Spatiotemporal Features via Video and Text Pair Discrimination
- Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video
- X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model
- A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval
- Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
- Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers
- Multimodal neural networks better explain multivoxel patterns in the hippocampus
- iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
- Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
- QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- AViD Dataset: Anonymized Videos from Diverse Countries
- A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
- CoVR-2: Automatic Data Construction for Composed Video Retrieval
- The complementarity of a diverse range of deep learning features extracted from video content for video recommendation
- Omni-sourced Webly-supervised Learning for Video Recognition
- Multi-queue Momentum Contrast for Microvideo-Product Retrieval
- Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey
- RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search
- Learning Video Representations from Textual Web Supervision
- Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
- Learning English with Peppa Pig
- Weak Supervision and Referring Attention for Temporal-Textual Association Learning
- Multimodal Pretraining for Dense Video Captioning
- Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
- RareAct: A video dataset of unusual interactions
- Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
- Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
- Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor Cues
- Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
- Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
- Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models
- T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
- Procedure Planning in Instructional Videos
- The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
- Expert Training: Task Hardness Aware Meta-Learning for Few-Shot Classification
- ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition
- Do Language Models Understand Time?
- Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval
- Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval
- Intra- and Inter-Action Understanding via Temporal Action Parsing
- Collaborative Three-Stream Transformers for Video Captioning
- ChinaOpen: A Dataset for Open-world Multimodal Learning
- POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World
- Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
- Video-Text Pre-training with Learned Regions
- VideoMix: Aggregating How-To Videos for Task-Oriented Learning
- Mixture of Experts for Decentralized Generative AI and Reinforcement Learning in Wireless Networks: A Comprehensive Survey
- Support-Set Based Cross-Supervision for Video Grounding
- A CLIP-Enhanced Method for Video-Language Understanding
- CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
- Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
- STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
- VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
- Semantic Role Aware Correlation Transformer for Text to Video Retrieval
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- Relevance-based Margin for Contrastively-trained Video Retrieval Models
- Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
- Universal Prototype Transport for Zero-Shot Action Recognition and Localization
- Disambiguation of weak supervision with exponential convergence rates
- IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval
- TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
- Audio-Language Datasets of Scenes and Events: A Survey
- Joint Multimedia Event Extraction from Video and Article
- What is More Likely to Happen Next? Video-and-Language Future Event Prediction
- Unsupervised Discovery of Actions in Instructional Videos
- Multi-modal Representation Learning for Video Advertisement Content Structuring
- Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval
- Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
- A Multi-level Alignment Training Scheme for Video-and-Language Grounding
- Survey: Transformer based Video-Language Pre-training
- Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
- PRVR: Partially Relevant Video Retrieval
- Learning Temporal Dynamics from Cycles in Narrated Video
- COBE: Contextualized Object Embeddings from Narrated Instructional Video
- ViSeRet: A simple yet effective approach to moment retrieval via fine-grained video segmentation
- Linguistic Structures as Weak Supervision for Visual Scene Graph Generation
- Multi-Granularity Network with Modal Attention for Dense Affective Understanding
- SSAN: Separable Self-Attention Network for Video Representation Learning
- Routing with Self-Attention for Multimodal Capsule Networks
- Multimedia-Aware Question Answering: A Review of Retrieval and Cross-Modal Reasoning Architectures
- What Matters for Ad-hoc Video Search? A Large-scale Evaluation on TRECVID
- Designing Multimodal Datasets for NLP Challenges
- CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
- Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning
- ActBERT: Learning Global-Local Video-Text Representations
- VILT: Video Instructions Linking for Complex Tasks
- SoccerHigh: A Benchmark Dataset for Automatic Soccer Video Summarization
- When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs
- Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision
- GEM: A General Evaluation Benchmark for Multimodal Tasks
- Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
- Video-aided Unsupervised Grammar Induction
- Supervision Levels Scale (SLS)
- Reconstructing and grounding narrated instructional videos in 3D
- YouMakeup VQA Challenge: Towards Fine-grained Action Understanding in Domain-Specific Videos
- CLIP4Caption: CLIP for Video Caption
- Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset
- Multi-modal Transformer for Video Retrieval
- CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
- SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval
- Cascaded Multilingual Audio-Visual Learning from Videos