4 citations · 5 across the 28 of their papers we have counts for
12 papers · 1 filter
MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
Yu Zhang, Hui-Ling Zhen, Mingxuan Yuan +1
Training large language models with FP8 formats offers significant efficiency gains. However, the reduced numerical precision of FP8 poses challenges for stable and accurate traini…
Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreak
Zixuan Huang, Kecheng Huang, Lihao Yin +4
Large Language Models (LLMs) have gained significant traction in various applications, yet their capabilities present risks for both constructive and malicious exploitation. Despit…
TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
Weizhe Lin, Xing Li, Zhiyuan Yang +7
Large Reasoning Models (LRMs) demonstrate exceptional capability in tackling complex mathematical, logical, and coding tasks by leveraging extended Chain-of-Thought (CoT) reasoning…
PreMoE: Proactive Inference for Efficient Mixture-of-Experts
Zehua Pei, Ying Zhang, Hui-Ling Zhen +6
Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization.…
A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources
Qingyu Song, Rui Liu, Wei Lin +11
Deploying Large Language Models (LLMs) on edge devices enhances privacy but faces performance hurdles due to limited resources. We introduce a systematic methodology to evaluate on…
SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention
Hong Yankun, Li Xing, Zhen Hui-Ling +3
For the efficient inference of Large Language Models (LLMs), the effective compression of key-value (KV) cache is essential. Three main types of KV cache compression techniques, na…