Irrational Complex Rotations Empower Low-bit Optimizers

Zhen Tian, Wayne Xin Zhao, Ji-Rong Wen

In this paper, we propose a novel optimizer state compression algorithm, namely $\pi$-Quant, which leverages the properties of irrational numbers (e.g., $\pi$) for memory-efficient tra…