8 papers
CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding
Aofan Liu, Jingxiang Meng, Fangxin Liu +1
Speculative decoding mitigates the latency of sequential generation in autoregressive Large Language Models (LLMs) by interleaving draft generation with target verification. Howeve…
COSM: A Cooperative Scheduling Framework for Concurrent PIM and CPU Execution on Mobile Devices
Yilong Zhao, Fangxin Liu, Onur Mutlu +4
The development of on-device large language models (LLMs) is driven by the need for privacy and fast response times. Energy-intensive data transfer on mobile devices makes Processi…
Accelerating Sparse Transformer Inference on GPU
Wenhao Dai, Haodong Deng, Mengfei Rong +6
Large language models (LLMs) are popular around the world due to their powerful understanding capabilities. As the core component of LLMs, accelerating Transformer through parallel…
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference
Xuwen Zhou, Fangxin Liu, Chao Wang +5
Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, part…
The Phantom of PCIe: Constraining Generative Artificial Intelligences for Practical Peripherals Trace Synthesizing
Zhibai Huang, Chen Chen, James Yen +9
Peripheral Component Interconnect Express (PCIe) is the de facto interconnect standard for high-speed peripherals and CPUs. The development of PCIe devices for emerging application…
HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures
Fangxin Liu, Qinghua Zhang, Hanjing Shen +5
The rapid evolution of Large Language Models (LLMs) towards long-context reasoning and sparse architectures has pushed memory requirements far beyond the capacity of individual dev…