Learning visual groups from co-occurrences in space and time

Abstract

We propose a self-supervised framework that learns to group visual entities based on their rate of co-occurrence in space and time. To model statistical dependencies between the entities, we set up a simple binary classification problem in which the goal is to predict if two visual primitives occur in the same spatial or temporal context. We apply this framework to three domains: learning patch affinities from spatial adjacency in images, learning frame affinities from temporal adjacency in videos, and learning photo affinities from geospatial proximity in image collections. We demonstrate that in each case the learned affinities uncover meaningful semantic groupings. From patch affinities we generate object proposals that are competitive with state-of-the-art supervised methods. From frame affinities we generate movie scene segmentations that correlate well with DVD chapter structure. Finally, from geospatial affinities we learn groups that relate well to semantic place categories.

References in corpus (4)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unsupervised Visual Representation Learning by Context Prediction
• Learning to See by Moving
• Learning image representations tied to ego-motion

Cited by in corpus (28)

• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Swapping Autoencoder for Deep Image Manipulation
• Contrastive Learning for Unpaired Image-to-Image Translation
• Space-Time Correspondence as a Contrastive Random Walk
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Colorization as a Proxy Task for Visual Understanding
• Neural Expectation Maximization
• On the Binding Problem in Artificial Neural Networks
• Video Representation Learning by Dense Predictive Coding
• Unsupervised Representation Learning by Sorting Sequences
• Learning Image Representations by Completing Damaged Jigsaw Puzzles
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
• MAST: A Memory-Augmented Self-supervised Tracker
• Self-Supervised Relative Depth Learning for Urban Scene Understanding
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• Self-Supervised Learning for Fine-Grained Visual Categorization
• Instance Separation Emerges from Inpainting
• Long-Short Temporal Contrastive Learning of Video Transformers
• Deep Unsupervised Clustering with Clustered Generator Model
• Improving Visual Recognition using Ambient Sound for Supervision
• Video Autoencoder: self-supervised disentanglement of static 3D structure and motion
• Unsupervised Image Segmentation by Mutual Information Maximization and Adversarial Regularization
• Learning Rich Representations For Structured Visual Prediction Tasks