Discovering Neural Wirings

Abstract

The success of neural networks has driven a shift in focus from feature engineering to architecture engineering. However, successful networks today are constructed using a small and manually defined set of building blocks. Even in methods of neural architecture search (NAS) the network connectivity patterns are largely constrained. In this work we propose a method for discovering neural wirings. We relax the typical notion of layers and instead enable channels to form connections independent of each other. This allows for a much larger space of possible networks. The wiring of our network is not fixed during training -- as we learn the network parameters we also learn the structure itself. Our experiments demonstrate that our learned connectivity outperforms hand engineered and randomly wired networks. By learning the connectivity of MobileNetV1we boost the ImageNet accuracy by 10% at ~41M FLOPs. Moreover, we show that our method generalizes to recurrent and continuous time networks. Our work may also be regarded as unifying core aspects of the neural architecture search problem with sparse neural network learning. As NAS becomes more fine grained, finding a good architecture is akin to finding a sparse subnetwork of the complete graph. Accordingly, DNW provides an effective mechanism for discovering sparse subnetworks of predefined architectures in a single training run. Though we only ever use a small percentage of the weights during the forward pass, we still play the so-called initialization lottery with a combinatorial number of subnetworks. Code and pretrained models are available at https://github.com/allenai/dnw while additional visualizations may be found at https://mitchellnw.github.io/blog/2019/dnw/.

NeurIPS 2019 Camera Ready

References in corpus (13)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Variational Dropout Sparsifies Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• Learning Sparse Networks Using Targeted Dropout
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

Cited by in corpus (19)

• Rigging the Lottery: Making All Tickets Winners
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Winning the Lottery with Continuous Sparsification
• Neural Architecture Generator Optimization
• Network Pruning That Matters: A Case Study on Retraining Variants
• FreezeNet: Full Performance by Reduced Storage Costs
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• NASGEM: Neural Architecture Search via Graph Embedding Method
• Differentiable Sparsification for Deep Neural Networks
• Searching for Stage-wise Neural Graphs In the Limit
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Lottery Jackpots Exist in Pre-trained Models
• Mining the Weights Knowledge for Optimizing Neural Network Structures
• AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
• Class-dependent Compression of Deep Neural Networks
• Understanding the wiring evolution in differentiable neural architecture search
• Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines
• Revisiting Neural Architecture Search
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks