Arithmetic Simplicity in Stochastic Gradient Methods

Abstract

A gradient descent method is arithmetically simple if the operations are limited to $+,-,\times$, and division $x/2^t$ with integer $t$. An arthmetically simple gradient method is easy to implement in chip design. We show how to transform AdaGrad, Adam, and AdamW into arithmetically simple. AdamW is based on the recursion $x_{t+1}=(1-\lambda \eta )x_t-\frac{\eta }{s}{m}_t$ and Adam is the special case of AdamW with $\lambda =0$. We transform them into a static case with $s=S(T)$, where $T$ is the number of iterations, and $S(T)$ is a fixed function. The convergence analysis is given for a static Adam, which is also arithmetically simple.