Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search

Abstract

Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. However, recent works have empirically shown a ranking disorder between the performance of stand-alone architectures and that of the corresponding shared-weight networks. This violates the main assumption of weight-sharing NAS algorithms, thus limiting their effectiveness. We tackle this issue by proposing a regularization term that aims to maximize the correlation between the performance rankings of the shared-weight network and that of the standalone architectures using a small set of landmark architectures. We incorporate our regularization term into three different NAS algorithms and show that it consistently improves performance across algorithms, search-spaces, and tasks.

arXiv commentAccepted to CVPR 2021

References in corpus (8)

• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Recurrent U-Net for Resource-Constrained Segmentation
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• Can weight sharing outperform random architecture search? An investigation with TuNAS