Memory-augmented Dense Predictive Coding for Video Representation Learning

Abstract

The objective of this paper is self-supervised learning from video, in particular for representations for action recognition. We make the following contributions: (i) We propose a new architecture and learning framework Memory-augmented Dense Predictive Coding (MemDPC) for the task. It is trained with a predictive attention mechanism over the set of compressed memories, such that any future states can always be constructed by a convex combination of the condense representations, allowing to make multiple hypotheses efficiently. (ii) We investigate visual-only self-supervised video representation learning from RGB frames, or from unsupervised optical flow, or both. (iii) We thoroughly evaluate the quality of learnt representation on four different downstream tasks: action recognition, video retrieval, learning with scarce annotations, and unintentional action classification. In all cases, we demonstrate state-of-the-art or comparable performance over other approaches with orders of magnitude fewer training data.

ECCV2020, Spotlight

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Unsupervised Learning of Visual Representations using Videos
• Non-local Neural Networks
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Learning to See by Moving
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Self-supervised learning of a facial attribute embedding from video
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Video Representation Learning by Dense Predictive Coding
• What have we learned from deep representations for action recognition?
• Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
• MAST: A Memory-Augmented Self-supervised Tracker
• Self-Supervised Video Representation Learning With Odd-One-Out Networks

Cited by in corpus (27)

• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• TCLR: Temporal Contrastive Learning for Video Representation
• A Comprehensive Study of Deep Video Action Recognition
• Spatiotemporal Contrastive Video Representation Learning
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
• Can Temporal Information Help with Contrastive Self-Supervised Learning?
• Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning
• Interactive Fusion of Multi-level Features for Compositional Activity Recognition
• Self-supervised Motion Learning from Static Images
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• Environment Predictive Coding for Embodied Agents
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
• Motion-Focused Contrastive Learning of Video Representations
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Learning Temporal Dynamics from Cycles in Narrated Video
• Addressing Feature Suppression in Unsupervised Visual Representations
• Video Contrastive Learning with Global Context
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• Unsupervised Visual Representation Learning by Tracking Patches in Video
• Towards Unified Surgical Skill Assessment
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• Long Short View Feature Decomposition via Contrastive Video Representation Learning
• Unsupervised Few-Shot Action Recognition via Action-Appearance Aligned Meta-Adaptation
• Learning Goals from Failure