MaskConnect: Connectivity Learning by Gradient Descent

Abstract

Although deep networks have recently emerged as the model of choice for many computer vision problems, in order to yield good results they often require time-consuming architecture search. To combat the complexity of design choices, prior work has adopted the principle of modularized design which consists in defining the network in terms of a composition of topologically identical or similar building blocks (a.k.a. modules). This reduces architecture search to the problem of determining the number of modules to compose and how to connect such modules. Again, for reasons of design complexity and training cost, previous approaches have relied on simple rules of connectivity, e.g., connecting each module to only the immediately preceding module or perhaps to all of the previous ones. Such simple connectivity rules are unlikely to yield the optimal architecture for the given problem. In this work we remove these predefined choices and propose an algorithm to learn the connections between modules in the network. Instead of being chosen a priori by the human designer, the connectivity is learned simultaneously with the weights of the network by optimizing the loss function of the end task using a modified version of gradient descent. We demonstrate our connectivity learning method on the problem of multi-class image classification using two popular architectures: ResNet and ResNeXt. Experiments on four different datasets show that connectivity learning using our approach yields consistently higher accuracy compared to relying on traditional predefined rules of connectivity. Furthermore, in certain settings it leads to significant savings in number of parameters.

arXiv commentECCV 2018. arXiv admin note: substantial text overlap with arXiv:1709.09582

References in corpus (24)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Matching Networks for One Shot Learning
• Learning both Weights and Connections for Efficient Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Scalable Bayesian Optimization Using Deep Neural Networks
• Shake-Shake regularization
• Hierarchical Representations for Efficient Architecture Search
• Convolutional Neural Fabrics
• Conditional Computation in Neural Networks for faster models
• Dynamic Network Surgery for Efficient DNNs
• Learning Factored Representations in a Deep Mixture of Experts
• Deep Learning of Representations: Looking Forward
• Learning the Structure of Deep Sparse Graphical Models
• Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
• Deep Sequential Neural Network
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning

Cited by in corpus (19)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Evaluating the Search Phase of Neural Architecture Search
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• AutoGAN: Neural Architecture Search for Generative Adversarial Networks
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Multi-Fiber Networks for Video Recognition
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search
• Variational Depth Search in ResNets
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks
• Learning Connectivity of Neural Networks from a Topological Perspective
• Revisiting Neural Architecture Search
• SparseMask: Differentiable Connectivity Learning for Dense Image Prediction