YouTube-8M: A Large-Scale Video Classification Benchmark

Abstract

Many recent advancements in Computer Vision are attributed to large datasets. Open-source software packages for Machine Learning and inexpensive commodity hardware have reduced the barrier of entry for exploring novel approaches at scale. It is possible to train models over millions of examples within a few days. Although large-scale datasets exist for image understanding, such as ImageNet, there are no comparable size video classification datasets. In this paper, we introduce YouTube-8M, the largest multi-label video classification dataset, composed of ~8 million videos (500K hours of video), annotated with a vocabulary of 4800 visual entities. To get the videos and their labels, we used a YouTube video annotation system, which labels videos with their main topics. While the labels are machine-generated, they have high-precision and are derived from a variety of human-based signals including metadata and query click signals. We filtered the video labels (Knowledge Graph entities) using both automated and manual curation strategies, including asking human raters if the labels are visually recognizable. Then, we decoded each video at one-frame-per-second, and used a Deep CNN pre-trained on ImageNet to extract the hidden representation immediately prior to the classification layer. Finally, we compressed the frame features and make both the features and video-level labels available for download. We trained various (modest) classification models on the dataset, evaluated them using popular evaluation metrics, and report them as baselines. Despite the size of the dataset, some of our models train to convergence in less than a day on a single machine using TensorFlow. We plan to release code for training a TensorFlow model and for computing metrics.

10 pages

References in corpus (7)

• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Visualizing and Understanding Convolutional Networks
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• Large-scale Multi-label Learning with Missing Labels
• Do Less and Achieve More: Training CNNs for Action Recognition Utilizing Action Images from the Web

Cited by in corpus (242)

• Media Forensics and DeepFakes: an overview
• Human Action Recognition from Various Data Modalities: A Review
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• FearNet: Brain-Inspired Model for Incremental Learning
• A Review on Deep Learning Techniques for Video Prediction
• YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
• YouTube UGC Dataset for Video Compression Research
• SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos
• Learnable pooling with Context Gating for video classification
• Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects
• Data-Free Knowledge Distillation for Deep Neural Networks
• BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
• Self-Supervised MultiModal Versatile Networks
• Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• TCLR: Temporal Contrastive Learning for Video Representation
• Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications
• A Comprehensive Study of Deep Video Action Recognition
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Review: Deep Learning in Electron Microscopy
• Video Background Music Generation with Controllable Music Transformer
• Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging
• Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• Spatiotemporal Contrastive Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
• NSML: A Machine Learning Platform That Enables You to Focus on Your Models
• Learning to Anticipate Egocentric Actions by Imagination
• BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
• SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries
• MERLOT: Multimodal Neural Script Knowledge Models
• Dataset Condensation with Differentiable Siamese Augmentation
• A Multi-viewpoint Outdoor Dataset for Human Action Recognition
• Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
• VideoMix: Rethinking Data Augmentation for Video Classification
• Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video
• VideoGraph: Recognizing Minutes-Long Human Activities in Videos
• VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views
• YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Measurement in AI Policy: Opportunities and Challenges
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition
• Predictive-Corrective Networks for Action Detection
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks
• Boundless: Generative Adversarial Networks for Image Extension
• The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks
• Single-frame Regularization for Temporally Stable CNNs
• Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers
• Graph-RISE: Graph-Regularized Image Semantic Embedding
• Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency
• Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network
• Unsupervised Representation Learning by Sorting Sequences
• Divide, Denoise, and Defend against Adversarial Attacks
• Decentralized Learning of Generative Adversarial Networks from Non-iid Data
• Neural Language Modeling with Visual Features
• Video action detection by learning graph-based spatio-temporal interactions
• SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
• Video-to-Music Recommendation using Temporal Alignment of Segments
• The Monkeytyping Solution to the YouTube-8M Video Understanding Challenge
• Deep Learning for Vision-based Prediction: A Survey
• YouTube-VOS: Sequence-to-Sequence Video Object Segmentation
• AViD Dataset: Anonymized Videos from Diverse Countries
• Task-Aware Variational Adversarial Active Learning
• Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
• The complementarity of a diverse range of deep learning features extracted from video content for video recommendation
• Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition
• M2m: Imbalanced Classification via Major-to-minor Translation
• A Single Video Super-Resolution GAN for Multiple Downsampling Operators based on Pseudo-Inverse Image Formation Models
• Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events
• Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms
• Feature Re-Learning with Data Augmentation for Video Relevance Prediction
• Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition
• Truly Multi-modal YouTube-8M Video Classification with Video, Audio, and Text
• Multi-queue Momentum Contrast for Microvideo-Product Retrieval
• On Real-time Image Reconstruction with Neural Networks for MRI-guided Radiotherapy
• Deep Audio-Visual Learning: A Survey
• Learning Video Representations from Textual Web Supervision
• Transfer Learning for the Efficient Detection of COVID-19 from Smartphone Audio Data
• Large Scale Organization and Inference of an Imagery Dataset for Public Safety
• Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification
• Deep Learning Methods for Efficient Large Scale Video Labeling
• Towards Universal Representation for Unseen Action Recognition
• Multigrid Predictive Filter Flow for Unsupervised Learning on Videos
• A Read-Write Memory Network for Movie Story Understanding
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• The IDLAB VoxCeleb Speaker Recognition Challenge 2020 System Description
• Aggregating Frame-level Features for Large-Scale Video Classification
• Moviescope: Large-scale Analysis of Movies using Multiple Modalities
• Memory Warps for Learning Long-Term Online Video Representations
• Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events
• Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition
• What Looks Good with my Sofa: Multimodal Search Engine for Interior Design
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation
• Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Content-Based Video-Music Retrieval Using Soft Intra-Modal Structure Constraint
• Unsupervised Learning from Video with Deep Neural Embeddings
• DramaQA: Character-Centered Video Story Understanding with Hierarchical QA
• Bayesian active learning for production, a systematic study and a reusable library
• Key Protected Classification for Collaborative Learning
• Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN
• A multimodal deep learning architecture for smoking detection with a small data approach
• Competitive Analysis System for Theatrical Movie Releases Based on Movie Trailer Deep Video Representation
• Cultivating DNN Diversity for Large Scale Video Labelling
• Multimodal Video-based Apparent Personality Recognition Using Long Short-Term Memory and Convolutional Neural Networks
• COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
• MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing
• Multi-Modal Detection of Alzheimer's Disease from Speech and Text
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Non-local NetVLAD Encoding for Video Classification
• Collaborative Three-Stream Transformers for Video Captioning
• Bottom-Up Temporal Action Localization with Mutual Regularization
• FSD-10: A Dataset for Competitive Sports Content Analysis
• Self-Supervised Video Representation Learning With Odd-One-Out Networks
• Understanding YouTube Communities via Subscription-based Channel Embeddings
• Attention and Localization based on a Deep Convolutional Recurrent Model for Weakly Supervised Audio Tagging
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery
• Uncertainty aware audiovisual activity recognition using deep Bayesian variational inference
• Relational Action Forecasting
• End-to-End Video Classification with Knowledge Graphs
• Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
• Large-scale Video Classification guided by Batch Normalized LSTM Translator
• Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• TinyAction Challenge: Recognizing Real-world Low-resolution Activities in Videos
• Self-Supervised Learning via Conditional Motion Propagation
• Multi-shot Temporal Event Localization: a Benchmark
• Less is More: Learning Highlight Detection from Video Duration
• Pillar Networks++: Distributed non-parametric deep and wide networks
• Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images
• Learning long-term dependencies for action recognition with a biologically-inspired deep network
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
• NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification
• DeepRemaster: Temporal Source-Reference Attention Networks for Comprehensive Video Enhancement
• BAR: Bayesian Activity Recognition using variational inference
• A Multimodal CNN-based Tool to Censure Inappropriate Video Scenes
• Cross-Class Relevance Learning for Temporal Concept Localization
• Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval
• Learning from Videos with Deep Convolutional LSTM Networks
• A Compressive Sensing Video dataset using Pixel-wise coded exposure
• Understanding and Training Deep Diagonal Circulant Neural Networks
• Large-Scale YouTube-8M Video Understanding with Deep Neural Networks
• Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning
• Spatio-Temporal Instance Learning: Action Tubes from Class Supervision
• Efficient Video Classification Using Fewer Frames
• ActionVLAD: Learning spatio-temporal aggregation for action classification
• Privacy-Preserving Visual Learning Using Doubly Permuted Homomorphic Encryption
• Synthetic Defocus and Look-Ahead Autofocus for Casual Videography
• Hallucinating Optical Flow Features for Video Classification
• Two-Stream Region Convolutional 3D Network for Temporal Activity Detection
• Unsupervised Domain Adaptation for Learning Eye Gaze from a Million Synthetic Images: An Adversarial Approach
• Learning to Segment Actions from Observation and Narration
• Optimizing Kernel Machines using Deep Learning
• Cross-modal Learning for Multi-modal Video Categorization
• A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysis
• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
• Semi-supervised Learning using Adversarial Training with Good and Bad Samples
• Accoustate: Auto-annotation of IMU-generated Activity Signatures under Smart Infrastructure
• DynaDog+T: A Parametric Animal Model for Synthetic Canine Image Generation
• BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference
• Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval
• Relighting Images in the Wild with a Self-Supervised Siamese Auto-Encoder
• Sparse Factorization Layers for Neural Networks with Limited Supervision
• CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition
• Unsupervised Multi-label Dataset Generation from Web Data
• Detecting Forged Facial Videos using convolutional neural network
• BERT for Large-scale Video Segment Classification with Test-time Augmentation
• Symbiotic Attention with Privileged Information for Egocentric Action Recognition
• Unsupervised Learning of Audio Perception for Robotics Applications: Learning to Project Data to T-SNE/UMAP space
• Cricket stroke extraction: Towards creation of a large-scale cricket actions dataset
• Two-Stream Video Classification with Cross-Modality Attention
• V3C - a Research Video Collection
• Deep Architectures and Ensembles for Semantic Video Classification
• Spatio-Temporal Channel Correlation Networks for Action Classification
• I Have Seen Enough: A Teacher Student Network for Video Classification Using Fewer Frames
• Discriminatively Learned Hierarchical Rank Pooling Networks
• Hierarchical Label Inference for Video Classification
• UTS submission to Google YouTube-8M Challenge 2017
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• Learnable Subspace Clustering
• YouTube-8M Video Understanding Challenge Approach and Applications
• An Effective Way to Improve YouTube-8M Classification Accuracy in Google Cloud Platform
• Exploiting Temporal Coherence for Multi-modal Video Categorization
• A Dataset and Benchmarks for Multimedia Social Analysis
• Temporally Coherent Video Harmonization Using Adversarial Networks
• Where and When: Space-Time Attention for Audio-Visual Explanations
• When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs
• Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep Multi-Task Learning
• Multi-Granularity Network with Modal Attention for Dense Affective Understanding
• Multi-attention Networks for Temporal Localization of Video-level Labels
• Enhancing Transformer for Video Understanding Using Gated Multi-Level Attention and Temporal Adversarial Training
• Online Action Detection in Streaming Videos with Time Buffers
• Multi-Label Zero-Shot Human Action Recognition via Joint Latent Ranking Embedding
• Compositional Structure Learning for Sequential Video Data
• Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
• Video Ads Content Structuring by Combining Scene Confidence Prediction and Tagging
• Out the Window: A Crowd-Sourced Dataset for Activity Classification in Security Video
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• CrowdMOT: Crowdsourcing Strategies for Tracking Multiple Objects in Videos
• Current Challenges and Future Directions in Podcast Information Access
• Learnable Pooling Methods for Video Classification
• Enhancing Video Music Recommendation with Transformer-Driven Audio-Visual Embeddings
• Employ Multimodal Machine Learning for Content quality analysis
• Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA
• Handcrafted vs Deep Learning Classification for Scalable Video QoE Modeling
• Self-supervised Attention Model for Weakly Labeled Audio Event Classification
• Multi-modal Aggregation for Video Classification
• Efficient Modelling Across Time of Human Actions and Interactions
• Visualizing Semantic Structures of Sequential Data by Learning Temporal Dependencies
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data
• Real-Time Object Detection and Localization in Compressive Sensed Video on Embedded Hardware
• Adversarially Robust Frame Sampling with Bounded Irregularities
• Adaptive and Iteratively Improving Recurrent Lateral Connections
• Deep Multimodal Learning: An Effective Method for Video Classification
• SDCNet: Video Prediction Using Spatially-Displaced Convolution
• Model-guided Multi-path Knowledge Aggregation for Aerial Saliency Prediction
• Learning Visual Voice Activity Detection with an Automatically Annotated Dataset
• Training compact deep learning models for video classification using circulant matrices
• Exploring Temporal Information for Improved Video Understanding
• Automatic Realistic Music Video Generation from Segments of Youtube Videos
• Classifying Video based on Automatic Content Detection Overview
• Label Denoising with Large Ensembles of Heterogeneous Neural Networks
• Constrained-size Tensorflow Models for YouTube-8M Video Understanding Challenge
• Smoothed Gaussian Mixture Models for Video Classification and Recommendation
• Semi-supervised Fisher vector network
• Cross-modal Spectrum Transformation Network For Acoustic Scene classification
• Large-Scale Mapping of Human Activity using Geo-Tagged Videos
• Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping
• Recurrent Residual Module for Fast Inference in Videos
• Play Fair: Frame Attributions in Video Models
• Hierarchical Deep Recurrent Architecture for Video Understanding
• Audiovisual transfer learning for audio tagging and sound event detection
• Can't Fool Me: Adversarially Robust Transformer for Video Understanding
• Summarizing First-Person Videos from Third Persons' Points of Views