Large Scale Neural Architecture Search with Polyharmonic Splines

Abstract

Neural Architecture Search (NAS) is a powerful tool to automatically design deep neural networks for many tasks, including image classification. Due to the significant computational burden of the search phase, most NAS methods have focused so far on small, balanced datasets. All attempts at conducting NAS at large scale have employed small proxy sets, and then transferred the learned architectures to larger datasets by replicating or stacking the searched cells. We propose a NAS method based on polyharmonic splines that can perform search directly on large scale, imbalanced target datasets. We demonstrate the effectiveness of our method on the ImageNet22K benchmark[16], which contains 14 million images distributed in a highly imbalanced manner over 21,841 categories. By exploring the search space of the ResNet [23] and Big-Little Net ResNext [11] architectures directly on ImageNet22K, our polyharmonic splines NAS method designed a model which achieved a top-1 accuracy of 40.03% on ImageNet22K, an absolute improvement of 3.13% over the state of the art with similar global batch size [15].

References in corpus (9)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• DARTS: Differentiable Architecture Search
• Neural Architecture Transfer
• Graph HyperNetworks for Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• XNAS: Neural Architecture Search with Expert Advice
• Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents