Exploring Randomly Wired Neural Networks for Image Recognition

Abstract

Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.

Technical report

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Statistical mechanics of complex networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DARTS: Differentiable Architecture Search
• Structural Properties of the Caenorhabditis elegans Neuronal Network
• Neural Architecture Optimization
• Random Search and Reproducibility for Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search

Cited by in corpus (42)

• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• A Survey on Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• Quantum Optimization with a Novel Gibbs Objective Function and Ansatz Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• Discovering Neural Wirings
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• Sample-Efficient Neural Architecture Search by Learning Action Space
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
• XNAS: Neural Architecture Search with Expert Advice
• An Adaptive Random Path Selection Approach for Incremental Learning
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• One-Shot Neural Architecture Search via Compressive Sensing
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• Speedy Performance Estimation for Neural Architecture Search
• Search for Better Students to Learn Distilled Knowledge
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• MnasFPN: Learning Latency-aware Pyramid Architecture for Object Detection on Mobile Devices
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• Learning Architectures for Binary Networks
• IOS: Inter-Operator Scheduler for CNN Acceleration
• Hybrid Composition with IdleBlock: More Efficient Networks for Image Recognition
• NAS-Count: Counting-by-Density with Neural Architecture Search
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• Bag of Tricks for Neural Architecture Search
• Multi-Pass Transformer for Machine Translation
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• Reinforcement Learning with Chromatic Networks for Compact Architecture Search
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• Dynamic Routing Networks
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
• Class-dependent Compression of Deep Neural Networks
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Understanding the wiring evolution in differentiable neural architecture search
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging