2.6k citations · 3.1k across the 36 of their papers we have counts for
4 papers · 1 filter
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit +4
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across…
Acceleration in First Order Quasi-strongly Convex Optimization by ODE Discretization
Jingzhao Zhang, Suvrit Sra, Ali Jadbabaie
We study gradient-based optimization methods obtained by direct Runge-Kutta discretization of the ordinary differential equation (ODE) describing the movement of a heavy-ball under…
Why gradient clipping accelerates training: A theoretical justification for adaptivity
Jingzhao Zhang, Tianxing He, Suvrit Sra +1
We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from prac…
Intrinsic long range antiferromagnetic coupling in dilutely V doped CuInTe
Weiyi Gong, Ching-Him Leung, Chuen-Keung Sin +4
Despite the various magnetic orders mediated by superexchange mechanism, the existence of a long range antiferromagnetic (AFM) coupling is unknown. Based on DFT calculations, we di…