Understanding the wiring evolution in differentiable neural architecture search
arXiv:2009.01272
Abstract
Controversy exists on whether differentiable neural architecture search methods discover wiring topology effectively. To understand how wiring topology evolves, we study the underlying mechanism of several existing differentiable NAS frameworks. Our investigation is motivated by three observed searching patterns of differentiable NAS: 1) they search by growing instead of pruning; 2) wider networks are more preferred than deeper ones; 3) no edges are selected in bi-level optimization. To anatomize these phenomena, we propose a unified view on searching algorithms of existing frameworks, transferring the global optimization to local cost minimization. Based on this reformulation, we conduct empirical and theoretical analyses, revealing implicit inductive biases in the cost's assignment mechanism and evolution dynamics that cause the observed phenomena. These biases indicate strong discrimination towards certain topologies. To this end, we pose questions that future differentiable methods for neural wiring discovery need to confront, hoping to evoke a discussion and rethinking on how much bias has been enforced implicitly in existing NAS methods.
arXiv commentAISTATS 2021
References in corpus (14)
- Neural Architecture Search with Reinforcement Learning
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Efficient Neural Architecture Search via Parameter Sharing
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- SNAS: Stochastic Neural Architecture Search
- DARTS+: Improved Differentiable Architecture Search with Early Stopping
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Exploring Randomly Wired Neural Networks for Image Recognition
- Discovering Neural Wirings
- The Dynamics of Learning: A Random Matrix Approach