Capacity allocation analysis of neural networks: A tool for principled architecture design

Abstract

Designing neural network architectures is a task that lies somewhere between science and art. For a given task, some architectures are eventually preferred over others, based on a mix of intuition, experience, experimentation and luck. For many tasks, the final word is attributed to the loss function, while for some others a further perceptual evaluation is necessary to assess and compare performance across models. In this paper, we introduce the concept of capacity allocation analysis, with the aim of shedding some light on what network architectures focus their modelling capacity on, when used on a given task. We focus more particularly on spatial capacity allocation, which analyzes a posteriori the effective number of parameters that a given model has allocated for modelling dependencies on a given point or region in the input space, in linear settings. We use this framework to perform a quantitative comparison between some classical architectures on various synthetic tasks. Finally, we consider how capacity allocation might translate in non-linear settings.

arXiv comment25 pages, 15 figures

References in corpus (8)

• Neural Architecture Search with Reinforcement Learning
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• On the Expressive Power of Deep Neural Networks
• Efficient Neural Audio Synthesis
• On Characterizing the Capacity of Neural Networks using Algebraic Topology
• Capacity allocation through neural network layers

Cited by in corpus (3)

• Capacity allocation through neural network layers
• Scaling up deep neural networks: a capacity allocation perspective
• Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology