5 papers · 1 filter
LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
Amir Reza Mirzaei, Yuqiao Wen, Yanshuai Cao +1
Low-Rank Adaptation (LoRA) has become a popular technique for parameter-efficient fine-tuning of large language models (LLMs). In many real-world scenarios, multiple adapters are l…
Exploring Model Invariance with Discrete Search for Ultra-Low-Bit Quantization
Yuqiao Wen, Yanshuai Cao, Lili Mou
Large language models have been increasing in size due to their success in a wide range of applications. This calls for a pressing need to reduce memory usage to make them more acc…
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
The performance of neural networks improves when more parameters are used. However, the model sizes are constrained by the available on-device memory during training and inference.…
Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
Second-order optimization approaches like the generalized Gauss-Newton method are considered more powerful as they utilize the curvature information of the objective function with…
Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Yongchang Hao, Yanshuai Cao, Lili Mou
Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To…