4 papers
Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability
Mitchell Scott, Tianshi Xu, Ziyuan Tang +4
Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise. We analyze preconditioned SGD in the geometry induced b…
Joint stochastic localization and applications
Tom Alberts, Yiming Xu, Qiang Ye
Stochastic localization is a pathwise analysis technique that has emerged as a powerful tool in high-dimensional probability and sampling. In this work, we extend stochastic locali…
Preconditioning for Accelerated Gradient Descent Optimization and Regularization
Qiang Ye
Accelerated training algorithms, such as adaptive learning rates (or preconditioning) and various normalization methods, are widely used but not fully understood. When regularizati…
Compact Recurrent Transformer with Persistent Memory
Edison Mucllari, Zachary Daniels, David Zhang +1
The Transformer architecture has shown significant success in many language processing and visual tasks. However, the method faces challenges in efficiently scaling to long sequenc…