18 papers
D-Quant: Driftable Entropy Coding for KV Cache Quantization
Yi Su, Hong Liu, Guanghua Yu +1
The KV cache has become a major bottleneck in deploying LLMs, as its memory footprint grows linearly with sequence length and batch size, imposing substantial pressure on both memo…
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Yufei Xue, Lin Niu, Hong Liu +6
The quadratic cost of self-attention makes long-context inference prohibitively expensive, and proxy-based block-sparse attention has become a practical remedy. Existing methods ty…
HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better
Gengluo Li, Xingyu Wan, Shangpin Peng +20
We present HunyuanOCR-1.5, a lightweight end-to-end OCR-specialized vision-language model. HunyuanOCR unifies document parsing, text spotting, information extraction, text-image tr…
FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
Xianglong Yan, Hong Liu, Chengzhu Bao +4
Large language models (LLMs) achieve remarkable performance but are expensive to deploy due to their enormous size. FP4 quantization, with formats such as MXFP4 and NVFP4, offers a…
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
Hong Liu, Rui Cen, Junhan Shi +10
Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads…
PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
Hong Liu, Yuan Cheng, Lin Niu +5
Token-level sparse attention, as implemented by DeepSeek Sparse Attention (DSA) in production systems, makes the downstream attention efficient but shifts the bottleneck to the ind…
D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
Tianyu Liu, Yuhao Shen, Rui Cen +7
Speculative decoding accelerates large language model (LLM) inference without compromising output quality. Recent parallel drafting methods further improve single-request performan…
DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding
Jiebin Zhang, Zhenghan Yu, Song Liu +9
Block diffusion speculative decoding accelerates LLM inference by predicting all tokens within a block simultaneously for the target model to verify in parallel. Predicting an enti…
Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild
Mao Zheng, Zheng Li, Tao Chen +10
Hy-MT2 is a family of fast-thinking multilingual translation models designed for complex real-world scenarios. It includes three model sizes: 1.8B, 7B, and 30B-A3B (MoE), all of wh…
SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
Chengzhu Bao, Xianglong Yan, Zhiteng Li +3
NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models (LLMs), offering superior numerical fidelity with native hardware support. However, e…
AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression
Rui Cen, QiangQiang Hu, Hong Huang +10
This technical report introduces AngelSlim, a comprehensive and versatile toolkit for large model compression developed by the Tencent Hunyuan team. By consolidating cutting-edge a…
DAQ: Delta-Aware Quantization for Post-Training LLM Weight Compression
Xiaoming Yu, Shize Tang, Guanghua Yu +4
We introduce Delta-Aware Quantization (DAQ), a data-free post-training quantization framework that preserves the knowledge acquired during post-training. Standard quantization obje…
Stem: Rethinking Causal Information Flow in Sparse Attention
Lin Niu, Xin Luo, Linchuan Xie +4
The quadratic computational complexity of self-attention remains a fundamental bottleneck for scaling Large Language Models (LLMs) to long contexts, particularly during the pre-fil…
IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs
Yifan Tan, Yifu Sun, Shirui Huang +4
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities, yet they encounter significant computational bottlenecks due to the massive volume of visual tok…
Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification
Hong Huang, Decheng Wu, Qiangqiang Hu +5
The deployment of Large Language Models (LLMs) on resource-constrained edge devices is increasingly hindered by prohibitive memory and computational requirements. While ternary qua…
SpecExit: Accelerating Large Reasoning Model via Speculative Exit
Rubing Yang, Huajun Bai, Song Liu +7
Despite their strong performance on reasoning tasks, large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-en…
Tequila: Trapping-free Ternary Quantization for Large Language Models
Hong Huang, Decheng Wu, Rui Cen +7
Quantization techniques are essential for the deployment of Large Language Models (LLMs) on edge devices. However, prevailing methods often rely on mixed-precision multiplication t…
Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
Tencent Hunyuan Team, Ao Liu, Botong Zhou +248
As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS, a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It synergistically combines Mam…