Exploring Randomly Wired Neural Networks for Image Recognition

Abstract

Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.

Technical report

References in corpus (3)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning

Cited by in corpus (16)

• A Survey on Neural Architecture Search
• XNAS: Neural Architecture Search with Expert Advice
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• Search for Better Students to Learn Distilled Knowledge
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• IOS: Inter-Operator Scheduler for CNN Acceleration
• Hybrid Composition with IdleBlock: More Efficient Networks for Image Recognition
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• Multi-Pass Transformer for Machine Translation
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Unchain the Search Space with Hierarchical Differentiable Architecture Search