Dynamic Rank Adjustment for Accurate and Efficient Neural Network Training

Hyuntak Shin, Aecheon Jung, Sungeun Hong +1

Low-rank training methods reduce the number of trainable parameters by re-parameterizing the weights with matrix decompositions (e.g., singular value decomposition). However, enfor…