Representation Learning: A Review and New Perspectives

Abstract

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

References in corpus (13)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Natural Language Processing (almost) from Scratch
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Practical recommendations for gradient-based training of deep architectures
• Better Mixing via Deep Representations
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
• What Regularized Auto-Encoders Learn from the Data Generating Distribution
• Multi-column Deep Neural Networks for Image Classification
• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields
• From Machine Learning to Machine Reasoning
• Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders
• On Training Deep Boltzmann Machines

Cited by in corpus (36)

• Learning Human Pose Estimation Features with Convolutional Networks
• Knowledge Matters: Importance of Prior Information for Optimization
• Learning Disentangled Semantic Representation for Domain Adaptation
• Causality for Machine Learning
• Challenges in Representation Learning: A report on three machine learning contests
• What Regularized Auto-Encoders Learn from the Data Generating Distribution
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Learning to Linearize Under Uncertainty
• Discriminative Recurrent Sparse Auto-Encoders
• Unsupervised Feature Learning from Temporal Data
• Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations
• Deeply Coupled Auto-encoder Networks for Cross-view Classification
• A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries
• Regularization Learning Networks: Deep Learning for Tabular Datasets
• Development of a skateboarding trick classifier using accelerometry and machine learning
• Sparsey: Event Recognition via Deep Hierarchical Spare Distributed Codes
• Knowledge Consistency between Neural Networks and Beyond
• Object Recognition Using Deep Neural Networks: A Survey
• Sample Complexity Analysis for Learning Overcomplete Latent Variable Models through Tensor Methods
• Explicit Disentanglement of Appearance and Perspective in Generative Models
• Learning about learning by many-body systems
• Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloud
• Unsupervised Learning of Spatiotemporally Coherent Metrics
• On Weak Lensing Shape Noise
• Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning
• Rethinking Sampling Strategies for Unsupervised Person Re-identification
• KnowBias: A Novel AI Method to Detect Polarity in Online Content
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• Unsupervised Pretraining Encourages Moderate-Sparseness
• Discriminative Relational Topic Models
• Learning Calibratable Policies using Programmatic Style-Consistency
• Disentangled Neural Architecture Search
• Learning Representations of Hierarchical Slates in Collaborative Filtering
• Automatically Segmenting the Left Atrium from Cardiac Images Using Successive 3D U-Nets and a Contour Loss
• Theory of Generative Deep Learning : Probe Landscape of Empirical Error via Norm Based Capacity Control
• Joint Estimation of Image Representations and their Lie Invariants