Neural networks adapting to datasets: learning network size and topology

Abstract

We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a standard gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset. The obtained networks can also be trained from scratch and achieve virtually identical performance. We explore the properties of the network architectures for a number of datasets of varying difficulty observing systematic regularities. The obtained graphs can be therefore understood as encoding nontrivial characteristics of the particular classification tasks.

arXiv commentFixed blank page

References in corpus (12)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Communicability in complex networks
• Pruning Filters for Efficient ConvNets
• The State of Sparsity in Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Exploring Sparsity in Recurrent Neural Networks
• What is the State of Neural Network Pruning?
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• Complexity for deep neural networks and other characteristics of deep feature representations