Wider Networks Learn Better Features
arXiv:1909.11572
Abstract
Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state of a wide network contains more information about the inputs than that of a narrow network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuning the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width.
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Opening the Black Box of Deep Neural Networks via Information
- A Convergence Theory for Deep Learning via Over-Parameterization
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Scaling description of generalization with number of parameters in deep learning
- SGD Learns the Conjugate Kernel Class of the Network