Understanding Short-Range Memory Effects in Deep Neural Networks

Chengli Tan, Jiangshe Zhang, Junmin Liu

Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success o…