Memory-Efficient Adaptive Optimization

Rohan Anil, Vineet Gupta, Tomer Koren +1

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these meth…