4 papers · 1 filter
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
The performance of neural networks improves when more parameters are used. However, the model sizes are constrained by the available on-device memory during training and inference.…
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
Dongheng Li, Yongchang Hao, Lili Mou
Large language models have become increasingly popular and demonstrated remarkable performance in various natural language processing (NLP) tasks. However, these models are typical…
Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with…
Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Yongchang Hao, Yanshuai Cao, Lili Mou
Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To…