5 papers
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Yongchang Hao, Lili Mou
Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enf…
TokMem: One-Token Procedural Memory for Large Language Models
Zijun Wu, Yongchang Hao, Lili Mou
Large language models are typically controlled via prompts, which must be repeatedly re-processed for every new query and are difficult to reuse modularly. We introduce TokMem, a p…
Ultra-Low-Dimensional Prompt Tuning via Random Projection
Zijun Wu, Yongchang Hao, Lili Mou
Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parame…
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
The performance of neural networks improves when more parameters are used. However, the model sizes are constrained by the available on-device memory during training and inference.…
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
Dongheng Li, Yongchang Hao, Lili Mou
Large language models have become increasingly popular and demonstrated remarkable performance in various natural language processing (NLP) tasks. However, these models are typical…