A Hessian-informed hyperparameter optimization for differential learning rate

Shiyun Xu, Zhiqi Bu, Yiliang Zhang +1

Differential learning rate (DLR), a technique that applies different learning rates to different model parameters, has been widely used in deep learning and achieved empirical succ…