Learning credit assignment

Abstract

Deep learning has achieved impressive prediction accuracies in a variety of scientific and industrial domains. However, the nested non-linear feature of deep learning makes the learning highly non-transparent, i.e., it is still unknown how the learning coordinates a huge number of parameters to achieve a decision making. To explain this hierarchical credit assignment, we propose a mean-field learning model by assuming that an ensemble of sub-networks, rather than a single network, are trained for a classification task. Surprisingly, our model reveals that apart from some deterministic synaptic weights connecting two neurons at neighboring layers, there exist a large number of connections that can be absent, and other connections can allow for a broad distribution of their weight values. Therefore, synaptic connections can be classified into three categories: very important ones, unimportant ones, and those of variability that may partially encode nuisance factors. Therefore, our model learns the credit assignment leading to the decision, and predicts an ensemble of sub-networks that can accomplish the same task, thereby providing insights toward understanding the macroscopic behavior of deep learning through the lens of distinct roles of synaptic weights.

arXiv comment5 pages, 4 figures, a generalized BackProp proposed to learn credit assignment from an network ensemble perspective, to appear in Phys Rev Lett (2020)

References in corpus (10)

• Overcoming catastrophic forgetting in neural networks
• Variational Inference: A Review for Statisticians
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Spike and slab variable selection: Frequentist and Bayesian strategies
• The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
• Random active path model of deep neural networks with diluted binary synapses
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Bayesian Neural Networks at Finite Temperature

Cited by in corpus (7)

• Eight challenges in developing theory of intelligence
• Statistical mechanics of continual learning: variational principle and mean-field potential
• Ensemble perspective for understanding temporal credit assignment
• A Search for Annihilating Dark Matter in 47 Tucanae and Omega Centauri
• Data-driven effective model shows a liquid-like deep learning
• Measurement-based quantum computation using two-component BECs
• Meta predictive learning model of languages in neural circuits