Rethinking Neural Operations for Diverse Tasks
arXiv:2103.15798
Abstract
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users to discover the right neural operations given data from their specific domain. We introduce a search space of operations called XD-Operations that mimic the inductive bias of standard multi-channel convolutions while being much more expressive: we prove that it includes many named operations across multiple application areas. Starting with any standard backbone such as ResNet, we show how to transform it into a search space over XD-operations and how to traverse the space using a simple weight-sharing scheme. On a diverse set of tasks -- solving PDEs, distance prediction for protein folding, and music modeling -- our approach consistently yields models with lower error than baseline networks and often even lower error than expert-designed domain-specific approaches.
arXiv commentNeurIPS 2021
References in corpus (18)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- DGM: A deep learning algorithm for solving partial differential equations
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
- NAS evaluation is frustratingly hard
- Trellis Networks for Sequence Modeling
- R-Transformer: Recurrent Neural Network Enhanced Transformer
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search
- Multidimensional Butterfly Factorization
- ACDC: A Structured Efficient Linear Layer
- Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
- Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practice