4 papers · 1 filter
Do Neural Networks Lose Plasticity in a Gradually Changing World?
Tianhui Liu, Lili Mou
Continual learning has become a trending topic in machine learning. Recent studies have discovered an interesting phenomenon called loss of plasticity, referring to neural networks…
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
The performance of neural networks improves when more parameters are used. However, the model sizes are constrained by the available on-device memory during training and inference.…
Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Yongchang Hao, Yanshuai Cao, Lili Mou
Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To…
Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with…