11 papers
Acceptance-Aware Draft Model Training for Speculative Decoding
Tianhua Xia, Mugilan Ganesan, Yifei Feng +3
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model…
EPIC: A System Framework for Efficient Egocentric Perception on Embodied AR Glasses
Tianhua Xia, Haiyu Wang, Jiajing Zheng +2
Modern smart AR glasses are evolving into intelligent systems that support foundation model-based assistance through continuous perception of the user and surrounding environment.…
DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation
Zining Liu, Yunhai Hu, Tianhua Xia +4
Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-lang…
DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution
Yunhai Hu, Zining Liu, Xiangyang Yin +5
Speculative reasoning has recently been proposed as a means to accelerate reasoning-intensive generation in large multimodal models, but its effectiveness is often constrained by m…
CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts
Xiangyang Yin, Xingyu Liu, Tianhua Xia +5
Outliers have emerged as a fundamental bottleneck in preserving accuracy for low-precision large models, particularly within Mixture-of-Experts (MoE) architectures that are increas…
Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
Tianhua Xia, Sai Qian Zhang
Running Large Language Models (LLMs) on edge devices is crucial for reducing latency, improving real-time processing, and enhancing privacy. By performing inference directly on the…
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Yunhai Hu, Tianhua Xia, Zining Liu +6
Speculative decoding (SD) has emerged as a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language mode…
Foveated Instance Segmentation
Hongyi Zeng, Wenxuan Liu, Tianhua Xia +3
Instance segmentation is essential for augmented reality and virtual reality (AR/VR) as it enables precise object recognition and interaction, enhancing the integration of virtual…
HAAN: A Holistic Approach for Accelerating Normalization Operations in Large Language Models
Tianfan Peng, Jiajun Qin, Tianhua Xia +1
Large language models (LLMs) have revolutionized natural language processing (NLP) tasks by achieving state-of-the-art performance across a range of benchmarks. Central to the succ…
Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
Patrick Yubeaton, Tareq Mahmoud, Shehab Naga +8
As they become more capable, large language models (LLMs) have continued to rapidly increase in size. This has exacerbated the difficulty in running state of the art LLMs on small,…
Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference
Tianhua Xia, Sai Qian Zhang
The attention mechanism is a pivotal element within the transformer architecture, making a substantial contribution to its exceptional performance. Within this attention mechanism,…