Viewmaker Networks: Learning Views for Unsupervised Representation Learning

Abstract

Many recent methods for unsupervised representation learning train models to be invariant to different "views," or distorted versions of an input. However, designing these views requires considerable trial and error by human experts, hindering widespread adoption of unsupervised representation learning methods across domains and modalities. To address this, we propose viewmaker networks: generative models that learn to produce useful views from a given input. Viewmakers are stochastic bounded adversaries: they produce views by generating and then adding an ${\ell }_p$-bounded perturbation to the input, and are trained adversarially with respect to the main encoder network. Remarkably, when pretraining on CIFAR-10, our learned views enable comparable transfer accuracy to the well-tuned SimCLR augmentations -- despite not including transformations like cropping or color jitter. Furthermore, our learned views significantly outperform baseline augmentations on speech recordings (+9% points, on average) and wearable sensor data (+17% points). Viewmakers can also be combined with handcrafted views: they improve robustness to common image corruptions and can increase transfer performance in cases where handcrafted views are less explored. These results suggest that viewmakers may provide a path towards more general representation learning algorithms -- reducing the domain expertise and effort needed to pretrain on a much wider set of domains. Code is available at https://github.com/alextamkin/viewmaker.

arXiv commentICLR 2021

References in corpus (34)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Data Augmentation Generative Adversarial Networks
• AutoAugment: Learning Augmentation Policies from Data
• Adversarial Feature Learning
• What Makes for Good Views for Contrastive Learning?
• Certified Defenses against Adversarial Examples
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Large Scale Adversarial Representation Learning
• Generating Adversarial Examples with Adversarial Networks
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• Adversarial Training Can Hurt Generalization
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• A Bayesian Data Augmentation Approach for Learning Deep Models
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• On Mutual Information in Contrastive Learning for Visual Representations
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Learning perturbation sets for robust machine learning
• Adversarial Contrastive Estimation
• Tempered Adversarial Networks

Cited by in corpus (8)

• On the Opportunities and Risks of Foundation Models
• MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Temperature as Uncertainty in Contrastive Learning
• Saturn Platform: Foundation Model Operations and Generative AI for Financial Services
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• Domain-Agnostic Clustering with Self-Distillation
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs