Narrowing the Focus: Learned Optimizers for Pretrained Models

Gus Kristiansen, Mark Sandler, Andrey Zhmoginov +4

In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand…