Deep Networks from the Principle of Rate Reduction

Abstract

This work attempts to interpret modern deep (convolutional) networks from the principles of rate reduction and (shift) invariant classification. We show that the basic iterative gradient ascent scheme for optimizing the rate reduction of learned features naturally leads to a multi-layer deep network, one iteration per layer. The layered architectures, linear and nonlinear operators, and even parameters of the network are all explicitly constructed layer-by-layer in a forward propagation fashion by emulating the gradient scheme. All components of this "white box" network have precise optimization, statistical, and geometric interpretation. This principled framework also reveals and justifies the role of multi-channel lifting and sparse coding in early stage of deep networks. Moreover, all linear operators of the so-derived network naturally become multi-channel convolutions when we enforce classification to be rigorously shift-invariant. The derivation also indicates that such a convolutional network is significantly more efficient to construct and learn in the spectral domain. Our preliminary simulations and experiments indicate that so constructed deep network can already learn a good discriminative representation even without any back propagation training.

References in corpus (9)

• Scikit-learn: Machine Learning in Python
• Neural Architecture Search with Reinforcement Learning
• Fast Training of Convolutional Networks through FFTs
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• On Learning Sets of Symmetric Elements
• Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery
• A Nonconvex Approach for Exact and Efficient Multichannel Sparse Blind Deconvolution

Cited by in corpus (6)

• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Recent Advances of Continual Learning in Computer Vision: An Overview
• Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs
• Toward Better Generalization Bounds with Locally Elastic Stability
• Incremental Learning via Rate Reduction
• Understanding Neural Networks with Logarithm Determinant Entropy Estimator