EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization

Abstract

Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing approaches are relatively expensive as they need to compute second-order derivatives and store a longer computational graph. This cost prevents scaling them to larger network architectures. We present EvoGrad, a new approach to meta-learning that draws upon evolutionary techniques to more efficiently compute hypergradients. EvoGrad estimates hypergradient with respect to hyperparameters without calculating second-order gradients, or storing a longer computational graph, leading to significant improvements in efficiency. We evaluate EvoGrad on three substantial recent meta-learning applications, namely cross-domain few-shot learning with feature-wise transformations, noisy label learning with Meta-Weight-Net and low-resource cross-lingual learning with meta representation transformation. The results show that EvoGrad significantly improves efficiency and enables scaling meta-learning to bigger architectures such as from ResNet10 to ResNet34.

arXiv commentAccepted at NeurIPS 2021

References in corpus (16)

• Deep Residual Learning for Image Recognition
• DARTS: Differentiable Architecture Search
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• AutoAugment: Learning Augmentation Policies from Data
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• On First-Order Meta-Learning Algorithms
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Meta-Learning with Implicit Gradients
• Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
• Generalized Inner Loop Meta-Learning
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Stochastic Hyperparameter Optimization through Hypernetworks
• Dataset Distillation
• Flexible Dataset Distillation: Learn Labels Instead of Images
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning