Using learned optimizers to make models robust to input noise

Abstract

State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution. However, when models are tested on corrupted images (e.g. due to scale changes, translations, or shifts in brightness or contrast), performance degrades significantly. Here, we explore the possibility of meta-training a learned optimizer that can train image classification models such that they are robust to common image corruptions. Specifically, we are interested training models that are more robust to noise distributions not present in the training data. We find that a learned optimizer meta-trained to produce models which are robust to Gaussian noise trains models that are more robust to Gaussian noise at other scales compared to traditional optimizers like Adam. The effect of meta-training is more complicated when targeting a more general set of noise distributions, but led to improved performance on half of held-out corruption tasks. Our results suggest that meta-learning provides a novel approach for studying and improving the robustness of deep learning models.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• AutoAugment: Learning Augmentation Policies from Data
• Neural Optimizer Search with Reinforcement Learning
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Unsupervised Learning via Meta-Learning
• Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Evolved Policy Gradients
• Learned Optimizers that Scale and Generalize
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Intriguing Properties of Adversarial Examples

Cited by in corpus (8)

• Learning to Optimize: A Primer and A Benchmark
• learn2learn: A Library for Meta-Learning Research
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Using a thousand optimization tasks to learn hyperparameter search strategies
• PDA: Progressive Data Augmentation for General Robustness of Deep Neural Networks
• Training Learned Optimizers with Randomly Initialized Learned Optimizers
• Neural Fixed-Point Acceleration for Convex Optimization