3 papers
cs.LG2025
AMS-QUANT: Adaptive Mantissa Sharing for Floating-point Quantization
Mengtao Lv, Ruiqi Zhu, Xinyu Wang +1
Large language models (LLMs) have demonstrated remarkable capabilities in various kinds of tasks, while the billion or even trillion parameters bring storage and efficiency bottlen…
cs.CL2024
PEARL: Parallel Speculative Decoding with Adaptive Draft Length
Tianyu Liu, Yun Li, Qitan Lv +4
Speculative decoding (SD), where an extra draft model is employed to provide multiple draft tokens first, and then the original target model verifies these tokens in parallel, has…
cs.LG2023
E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity
Yun Li, Lin Niu, Xipeng Zhang +3
Traditional pruning methods are known to be challenging to work in Large Language Models (LLMs) for Generative AI because of their unaffordable training process and large computati…