most citedNoise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be

5 citations · 5 across the 3 of their papers we have counts for

collaborators
Showing math.OCShow all

1 paper · 1 filter

math.OC2023

Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking

Frederik Kunstner, Victor S. Portella, Mark Schmidt +1

The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically opti…