Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations

Abstract

Deep-predictive-coding networks (DPCNs) are hierarchical, generative models. They rely on feed-forward and feed-back connections to modulate latent feature representations of stimuli in a dynamic and context-sensitive manner. A crucial element of DPCNs is a forward-backward inference procedure to uncover sparse, invariant features. However, this inference is a major computational bottleneck. It severely limits the network depth due to learning stagnation. Here, we prove why this bottleneck occurs. We then propose a new forward-inference strategy based on accelerated proximal gradients. This strategy has faster theoretical convergence guarantees than the one used for DPCNs. It overcomes learning stagnation. We also demonstrate that it permits constructing deep and wide predictive-coding networks. Such convolutional networks implement receptive fields that capture well the entire classes of objects on which the networks are trained. This improves the feature representations compared with our lab's previous non-convolutional and convolutional DPCNs. It yields unsupervised object recognition that surpass convolutional autoencoders and are on par with convolutional networks trained in a supervised manner.

arXiv commentSubmitted to the IEEE Transactions on Neural Networks and Learning Systems

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Residual Learning for Image Recognition
• Striving for Simplicity: The All Convolutional Net
• Training Very Deep Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Learning Activation Functions to Improve Deep Neural Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• The rate of convergence of Nesterov's accelerated forward-backward method is actually faster than $1/k^2$
• Stacked What-Where Auto-encoders
• A sparse coding model with synaptically local plasticity and spiking neurons can account for the diverse shapes of V1 simple cell receptive fields
• Spectral Representations for Convolutional Neural Networks
• Convolutional Kernel Networks
• Training Neural Networks with Local Error Signals
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• Improving Deep Neural Networks with Probabilistic Maxout Units
• Sparse Codes for Speech Predict Spectrotemporal Receptive Fields in the Inferior Colliculus
• Adaptive restart of accelerated gradient methods under local quadratic growth condition
• Stacked Capsule Autoencoders
• Deep Predictive Coding Networks