4 papers
Do Neural Networks Lose Plasticity in a Gradually Changing World?
Tianhui Liu, Lili Mou
Continual learning has become a trending topic in machine learning. Recent studies have discovered an interesting phenomenon called loss of plasticity, referring to neural networks…
Ultra-Low-Dimensional Prompt Tuning via Random Projection
Zijun Wu, Yongchang Hao, Lili Mou
Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parame…
TokMem: One-Token Procedural Memory for Large Language Models
Zijun Wu, Yongchang Hao, Lili Mou
Large language models are typically controlled via prompts, which must be repeatedly re-processed for every new query and are difficult to reuse modularly. We introduce TokMem, a p…
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
The performance of neural networks improves when more parameters are used. However, the model sizes are constrained by the available on-device memory during training and inference.…