4 papers
AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression
Rui Cen, QiangQiang Hu, Hong Huang +10
This technical report introduces AngelSlim, a comprehensive and versatile toolkit for large model compression developed by the Tencent Hunyuan team. By consolidating cutting-edge a…
Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification
Hong Huang, Decheng Wu, Qiangqiang Hu +5
The deployment of Large Language Models (LLMs) on resource-constrained edge devices is increasingly hindered by prohibitive memory and computational requirements. While ternary qua…
Tequila: Trapping-free Ternary Quantization for Large Language Models
Hong Huang, Decheng Wu, Rui Cen +7
Quantization techniques are essential for the deployment of Large Language Models (LLMs) on edge devices. However, prevailing methods often rely on mixed-precision multiplication t…
NeCTAr: A Heterogeneous RISC-V SoC for Language Model Inference in Intel 16
Viansa Schmulbach, Jason Kim, Ethan Gao +4
This paper introduces NeCTAr (Near-Cache Transformer Accelerator), a 16nm heterogeneous multicore RISC-V SoC for sparse and dense machine learning kernels with both near-core and n…