21 citations · 33 across the 28 of their papers we have counts for
10 papers · 1 filter
MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
Yu Zhang, Hui-Ling Zhen, Mingxuan Yuan +1
Training large language models with FP8 formats offers significant efficiency gains. However, the reduced numerical precision of FP8 poses challenges for stable and accurate traini…
TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
Weizhe Lin, Xing Li, Zhiyuan Yang +7
Large Reasoning Models (LRMs) demonstrate exceptional capability in tackling complex mathematical, logical, and coding tasks by leveraging extended Chain-of-Thought (CoT) reasoning…
PreMoE: Proactive Inference for Efficient Mixture-of-Experts
Zehua Pei, Ying Zhang, Hui-Ling Zhen +6
Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization.…
A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources
Qingyu Song, Rui Liu, Wei Lin +11
Deploying Large Language Models (LLMs) on edge devices enhances privacy but faces performance hurdles due to limited resources. We introduce a systematic methodology to evaluate on…
SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention
Hong Yankun, Li Xing, Zhen Hui-Ling +3
For the efficient inference of Large Language Models (LLMs), the effective compression of key-value (KV) cache is essential. Three main types of KV cache compression techniques, na…
Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attacks
Bowei He, Lihao Yin, Hui-Ling Zhen +4
The widespread deployment of pre-trained language models (PLMs) has exposed them to textual backdoor attacks, particularly those planted during the pre-training stage. These attack…
KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
Xing Li, Zeyu Xing, Yiming Li +6
KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness.…
Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
Zehua Pei, Hui-Ling Zhen, Lancheng Zou +5
Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational res…
MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
Yu Zhang, Mingzi Wang, Lancheng Zou +4
Transformer-based large language models (LLMs) have achieved remarkable success as model sizes continue to grow, yet their deployment remains challenging due to significant computa…
From Pruning to Grafting: Dynamic Knowledge Redistribution via Learnable Layer Fusion
Zehua Pei, Hui-Ling Zhen, Xianzhi Yu +3
Structured pruning of Generative Pre-trained Transformers (GPTs) offers a promising path to efficiency but often suffers from irreversible performance degradation due to the discar…