First-Order Preconditioning via Hypergradient Descent

Ted Moskovitz, Rui Wang, Janice Lan +4

Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulti…