Learning with hidden variables

Abstract

Learning and inferring features that generate sensory input is a task continuously performed by cortex. In recent years, novel algorithms and learning rules have been proposed that allow neural network models to learn such features from natural images, written text, audio signals, etc. These networks usually involve deep architectures with many layers of hidden neurons. Here we review recent advancements in this area emphasizing, amongst other things, the processing of dynamical inputs by networks with hidden nodes and the role of single neuron models. These points and the questions they arise can provide conceptual advancements in understanding of learning in the cortex and the relationship between machine learning approaches to learning with hidden nodes and those in cortical circuits.

revised version accepted in Current Opinion in Neurobiology

References in corpus (18)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Improving neural networks by preventing co-adaptation of feature detectors
• Two-Stream Convolutional Networks for Action Recognition in Videos
• On the difficulty of training Recurrent Neural Networks
• Proceedings of the 29th International Conference on Machine Learning (ICML-12)
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Towards Biologically Plausible Deep Learning
• Learning Longer Memory in Recurrent Neural Networks
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
• Show and Tell: A Neural Image Caption Generator
• Mean Field Theory For Non-Equilibrium Network Reconstruction
• Belief-Propagation and replicas for inference and learning in a kinetic Ising model with hidden spins
• Inferring hidden states in a random kinetic Ising model: replica analysis

Cited by in corpus (8)

• Inverse statistical problems: from the inverse Ising problem to data science
• Towards an integration of deep learning and neuroscience
• Quantifying Relevance in Learning and Inference
• Inverse Ising problem in continuous time: A latent variable approach
• Correlation-Compressed Direct Coupling Analysis
• Improved Pseudolikelihood Regularization and Decimation methods on Non-linearly Interacting Systems with Continuous Variables
• Transfer entropy-based feedback improves performance in artificial neural networks
• A random energy approach to deep learning