Practical tradeoffs between memory, compute, and performance in learned optimizers

Luke Metz, C. Daniel Freeman, James Harrison +2

Optimization plays a costly and crucial role in developing machine learning systems. In learned optimizers, the few hyperparameters of commonly used hand-designed optimizers, e.g.…