Deep Learning of Representations: Looking Forward

Abstract

Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.

References in corpus (16)

• Improving neural networks by preventing co-adaptation of feature detectors
• Natural Language Processing (almost) from Scratch
• On the difficulty of training Recurrent Neural Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Deep Generative Stochastic Networks Trainable by Backprop
• Better Mixing via Deep Representations
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
• Conditional Restricted Boltzmann Machines for Structured Output Prediction
• Estimating or Propagating Gradients Through Stochastic Neurons
• Disentangling Factors of Variation via Generative Entangling
• Learning Feature Hierarchies with Centered Deep Boltzmann Machines
• Multi-column Deep Neural Networks for Image Classification
• Big Neural Networks Waste Capacity
• A Semantic Matching Energy Function for Learning with Multi-relational Data
• Evolving Culture vs Local Minima
• Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders

Cited by in corpus (51)

• Deep Learning for Sensor-based Activity Recognition: A Survey
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Predicting Parameters in Deep Learning
• Learning Features for Offline Handwritten Signature Verification using Deep Convolutional Neural Networks
• Recent Advances in Convolutional Neural Networks
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Learning Factored Representations in a Deep Mixture of Experts
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Recent Advances in Deep Learning: An Overview
• Domain Adaptive Neural Networks for Object Recognition
• Learning model-based planning from scratch
• MaskConnect: Connectivity Learning by Gradient Descent
• Neural Expectation Maximization
• BlockDrop: Dynamic Inference Paths in Residual Networks
• Deep Learning in Customer Churn Prediction: Unsupervised Feature Learning on Abstract Company Independent Feature Vectors
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Representing high throughput expression profiles via perturbation barcodes reveals compound targets
• SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
• Consistent Feature Construction with Constrained Genetic Programming for Experimental Physics
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Denoising autoencoder with modulated lateral connections learns invariant representations of natural images
• Why does Deep Learning work? - A perspective from Group Theory
• Self-informed neural network structure learning
• Learning Deep Representations for Semantic Image Parsing: a Comprehensive Overview
• Learning to Adapt by Minimizing Discrepancy
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Measuring relative opinion from location-based social media: A case study of the 2016 U.S. presidential election
• Dynamic Capacity Networks
• A Deep Neural Network for Chinese Zero Pronoun Resolution
• A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
• Exploring Uncertainty Measures for Image-Caption Embedding-and-Retrieval Task
• Deep Learning Application in Security and Privacy -- Theory and Practice: A Position Paper
• Recognizing Video Events with Varying Rhythms
• On the Equivalence Between Deep NADE and Generative Stochastic Networks
• Fullie and Wiselie: A Dual-Stream Recurrent Convolutional Attention Model for Activity Recognition
• Learning a Hierarchical Latent-Variable Model of 3D Shapes
• On the effect of pooling on the geometry of representations
• Learning Deep Representation Without Parameter Inference for Nonlinear Dimensionality Reduction
• Radical-Enhanced Chinese Character Embedding
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• AI Discovering a Coordinate System of Chemical Elements: Dual Representation by Variational Autoencoders
• Concepts, Properties and an Approach for Compositional Generalization
• Adversarial Concurrent Training: Optimizing Robustness and Accuracy Trade-off of Deep Neural Networks
• Feature Graph Architectures
• Towards Machine Intelligence
• Deep Generative Networks For Sequence Prediction
• Strongly-Typed Agents are Guaranteed to Interact Safely
• Channel selection using Gumbel Softmax
• Learning Term Weights for Ad-hoc Retrieval
• Deep Learning the Effects of Photon Sensors on the Event Reconstruction Performance in an Antineutrino Detector
• Generic Semi-Supervised Adversarial Subject Translation for Sensor-Based Human Activity Recognition