Using a thousand optimization tasks to learn hyperparameter search strategies

Abstract

We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets. As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as ~29 million training curves for these problems and the corresponding hyperparameters.

References in corpus (48)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Decoupled Weight Decay Regularization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• OpenML: networked science in machine learning
• Searching for Activation Functions
• Density estimation using Real NVP
• Emergence of Locomotion Behaviours in Rich Environments
• On the Variance of the Adaptive Learning Rate and Beyond
• DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning
• DeepMind Control Suite
• Scalable Bayesian Optimization Using Deep Neural Networks
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• The Natural Language Decathlon: Multitask Learning as Question Answering
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Neural Optimizer Search with Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• On Empirical Comparisons of Optimizers for Deep Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Using Fast Weights to Attend to the Recent Past
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• Freeze-Thaw Bayesian Optimization
• Learned Optimizers that Scale and Generalize
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Learning to Optimize Neural Nets
• Bayesian Optimization in AlphaGo
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Unit Tests for Stochastic Optimization
• Open-ended Learning in Symmetric Zero-sum Games
• Understanding and correcting pathologies in the training of learned optimizers
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• Critical Hyper-Parameters: No Random, No Cry
• A Stratified Analysis of Bayesian Optimization Methods
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• Using learned optimizers to make models robust to input noise
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

Cited by in corpus (9)

• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Parallel Training of Deep Networks with Local Updates
• Meta Learning Black-Box Population-Based Optimizers
• Hyperparameter Transfer Across Developer Adjustments
• Meta-Learning for Symbolic Hyperparameter Defaults
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers