5 papers
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Shenggui Li, Chao Wang, Yikai Zhu +14
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to…
First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models
Chi Ma, Mincong Huang, Ying Zhang +5
Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LL…
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
Ying Zhang, Peng Zhang, Mincong Huang +7
Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often…
MOYU: A Theoretical Study on Massive Over-activation Yielded Uplifts in LLMs
Chi Ma, Mincong Huang, Chao Wang +2
Massive Over-activation Yielded Uplifts(MOYU) is an inherent property of large language models, and dynamic activation(DA) based on the MOYU property is a clever yet under-explored…
Dynamic Activation Pitfalls in LLaMA Models: An Empirical Study
Chi Ma, Mincong Huang, Chao Wang +2
In this work, we systematically investigate the efficacy of dynamic activation mechanisms within the LLaMA family of language models. Despite the potential of dynamic activation me…