7 papers
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Liam Cooper, Shinnung Jeong, Hyeran Jeon +2
Large language model (LLM) outputs are expected to be reproducible under greedy decoding, yet in practice the same model, prompt, and software stack produce different outputs on di…
AutoUVM: Automated Prefetching Framework for LLMs under UVM Oversubscription
Mao Lin, Hui Feng, Xianzhong Ding +3
Large language models (LLMs) increasingly exceed the memory capacity of commodity GPUs, making memory oversubscription common in practical deployments. NVIDIA Unified Virtual Memor…
HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
Mao Lin, Xi Wang, Guilherme Cox +2
As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruni…
ROKA: Robust Knowledge Unlearning against Adversaries
Jinmyeong Shin, Joshua Tapia, Nicholas Ferreira +3
The need for machine unlearning is critical for data privacy, yet existing methods often cause Knowledge Contamination by unintentionally damaging related knowledge. Such a degrade…
PASTA: A Modular Program Analysis Tool Framework for Accelerators
Mao Lin, Hyeran Jeon, Keren Zhou
The increasing complexity and diversity of hardware accelerators in modern computing systems demand flexible, low-overhead program analysis tools. We present PASTA, a low-overhead…
AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
Dinghong Song, Yuan Feng, Yiwei Wang +6
Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification,…
ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNs
Fang Chen, Gourav Datta, Mujahid Al Rafi +2
The expansion of neural network sizes and the enhanced resolution of modern image sensors result in heightened memory and power demands to process modern computer vision models. In…