On the Duality between Gradient Transformations and Adapters

Lucas Torroba-Hennigen, Hunter Lang, Han Guo +1

We study memory-efficient optimization of neural networks (in particular language models) with linear gradient transformations, where the gradients are linearly mapped to a lower d…