Capacity allocation through neural network layers

Abstract

Capacity analysis has been recently introduced as a way to analyze how linear models distribute their modelling capacity across the input space. In this paper, we extend the notion of capacity allocation to the case of neural networks with non-linear layers. We show that under some hypotheses the problem is equivalent to linear capacity allocation, within some extended input space that factors in the non-linearities. We introduce the notion of layer decoupling, which quantifies the degree to which a non-linear activation decouples its outputs, and show that it plays a central role in capacity allocation through layers. In the highly non-linear limit where decoupling is total, we show that the propagation of capacity throughout the layers follows a simple markovian rule, which turns into a diffusion PDE in the limit of deep networks with residual layers. This allows us to recover some known results about deep neural networks, such as the size of the effective receptive field, or why ResNets avoid the shattering problem.

14 pages, 4 figures

References in corpus (6)

• Empirical Evaluation of Rectified Activations in Convolutional Network
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• On the Expressive Power of Deep Neural Networks
• Efficient Neural Audio Synthesis
• On Characterizing the Capacity of Neural Networks using Algebraic Topology
• Capacity allocation analysis of neural networks: A tool for principled architecture design

Cited by in corpus (2)

• Capacity allocation analysis of neural networks: A tool for principled architecture design
• Scaling up deep neural networks: a capacity allocation perspective