Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

JuneHyung Kim, Sankeerth Durvasula, Nandita Vijaykumar

Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded wei…