AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data

Abstract

The success of deep neural networks often relies on a large amount of labeled examples, which can be difficult to obtain in many real scenarios. To address this challenge, unsupervised methods are strongly preferred for training neural networks without using any labeled data. In this paper, we present a novel paradigm of unsupervised representation learning by Auto-Encoding Transformation (AET) in contrast to the conventional Auto-Encoding Data (AED) approach. Given a randomly sampled transformation, AET seeks to predict it merely from the encoded features as accurately as possible at the output end. The idea is the following: as long as the unsupervised features successfully encode the essential information about the visual structures of original and transformed images, the transformation can be well predicted. We will show that this AET paradigm allows us to instantiate a large variety of transformations, from parameterized, to non-parameterized and GAN-induced ones. Our experiments show that AET greatly improves over existing unsupervised approaches, setting new state-of-the-art performances being greatly closer to the upper bounds by their fully supervised counterparts on CIFAR-10, ImageNet and Places datasets.

in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, June 16th - June 20th, 2019

References in corpus (6)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Adversarial Feature Learning
• Adversarially Learned Inference
• Deep Clustering for Unsupervised Learning of Visual Features
• Data-dependent Initializations of Convolutional Neural Networks
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Cited by in corpus (14)

• Prototypical Contrastive Learning of Unsupervised Representations
• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• AVT: Unsupervised Learning of Transformation Equivariant Representations by Autoencoding Variational Transformations
• GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-wise Transformations
• FLAT: Few-Shot Learning via Autoencoding Transformation Regularizers
• Unsupervised Image Classification for Deep Representation Learning
• Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning
• Laplacian Denoising Autoencoder
• Reliable Label Bootstrapping for Semi-Supervised Learning
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations