Hyperparameter optimization with REINFORCE and Transformers

Abstract

Reinforcement Learning has yielded promising results for Neural Architecture Search (NAS). In this paper, we demonstrate how its performance can be improved by using a simplified Transformer block to model the policy network. The simplified Transformer uses a 2-stream attention-based mechanism to model hyper-parameter dependencies while avoiding layer normalization and position encoding. We posit that this parsimonious design balances model complexity against expressiveness, making it suitable for discovering optimal architectures in high-dimensional search spaces with limited exploration budgets. We demonstrate how the algorithm's performance can be further improved by a) using an actor-critic style algorithm instead of plain vanilla policy gradient and b) ensembling Transformer blocks with shared parameters, each block conditioned on a different auto-regressive factorization order. Our algorithm works well as both a NAS and generic hyper-parameter optimization (HPO) algorithm: it outperformed most algorithms on NAS-Bench-101, a public data-set for benchmarking NAS algorithms. In particular, it outperformed RL based methods that use alternate architectures to model the policy network, underlining the value of using attention-based networks in this setting. As a generic HPO algorithm, it outperformed Random Search in discovering more accurate multi-layer perceptron model architectures across 2 regression tasks. We have adhered to guidelines listed in Lindauer and Hutter while designing experiments and reporting results.

References in corpus (22)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• On the difficulty of training Recurrent Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Neural Architecture Search: A Survey
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Pixel Recurrent Neural Networks
• Character-Aware Neural Language Models
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• MADE: Masked Autoencoder for Distribution Estimation
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Hierarchical Representations for Efficient Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• Probabilistic Matrix Factorization for Automated Machine Learning
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• PixelSNAIL: An Improved Autoregressive Generative Model
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search