Building high-level features using large scale unsupervised learning

Abstract

We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 20,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art.

References in corpus (1)

• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields

Cited by in corpus (51)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Intriguing properties of neural networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Fully Convolutional Networks for Semantic Segmentation
• Unsupervised Visual Representation Learning by Context Prediction
• Event-Driven Contrastive Divergence for Spiking Neuromorphic Systems
• Unsupervised Learning of Visual Representations using Videos
• Symbol Emergence in Robotics: A Survey
• Joint Unsupervised Learning of Deep Representations and Image Clusters
• Medical Concept Representation Learning from Electronic Health Records and its Application on Heart Failure Prediction
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
• Unsupervised Learning of Invariant Representations in Hierarchical Architectures
• Learning by Association - A versatile semi-supervised training method for neural networks
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Design and Construction of a Brain-Like Computer: A New Class of Frequency-Fractal Computing Using Wireless Communication in a Supramolecular Organic, Inorganic System
• A Study on various state of the art of the Art Face Recognition System using Deep Learning Techniques
• Neuronal Synchrony in Complex-Valued Deep Networks
• Recognizing Semantic Features in Faces using Deep Learning
• Deep Convolutional Features for Image Based Retrieval and Scene Categorization
• Learnable Pooling Regions for Image Classification
• Unsupervised Representation Learning by Sorting Sequences
• PANDA: Pose Aligned Networks for Deep Attribute Modeling
• Flip-Rotate-Pooling Convolution and Split Dropout on Convolution Neural Networks for Image Classification
• Self-taught Object Localization with Deep Networks
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• A convolution recurrent autoencoder for spatio-temporal missing data imputation
• Differentiable Pooling for Hierarchical Feature Learning
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Variational Implicit Processes
• Unsupervised Deep Representation Learning for Real-Time Tracking
• Online Deep Clustering for Unsupervised Representation Learning
• Nonparametric Weight Initialization of Neural Networks via Integral Representation
• Deep Roto-Translation Scattering for Object Classification
• Deepened Graph Auto-Encoders Help Stabilize and Enhance Link Prediction
• Audio Classical Composer Identification by Deep Neural Network
• Survival of the Fittest in PlayerUnknown BattleGround
• Unsupervised Feature Learning with C-SVDDNet
• Learning from Noisy Labels with Noise Modeling Network
• DAFAR: Defending against Adversaries by Feedback-Autoencoder Reconstruction
• DeepCoder: Semi-parametric Variational Autoencoders for Automatic Facial Action Coding
• Training Deep Fourier Neural Networks To Fit Time-Series Data
• Perceive Where to Focus: Learning Visibility-aware Part-level Features for Partial Person Re-identification
• Rank Ordered Autoencoders
• Deep Extreme Feature Extraction: New MVA Method for Searching Particles in High Energy Physics
• Towards co-evolution of fitness predictors and Deep Neural Networks
• Multiple Kernel Learning and Automatic Subspace Relevance Determination for High-dimensional Neuroimaging Data
• Is 'Unsupervised Learning' a Misconceived Term?
• Feature and Variable Selection in Classification
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks