4 papers
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
Chao Fang, Jun Yin, Man Shi +1
With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this ch…
SparseCol: A 1320 BTOPS/W Precision-scalable NPU Exploiting Training-free Structured Bit-level Sparsity and Dynamic Dataflow
Man Shi, Vikram Jain, Weijie Jiang +4
Bit-serial computation enables sequential processing of data at the bit level, providing several advantages, such as scalable computational precision. This approach has gained sign…
BitWave: Exploiting Column-Based Bit-Level Sparsity for Deep Learning Acceleration
Man Shi, Vikram Jain, Antony Joseph +2
Bit-serial computation facilitates bit-wise sequential data processing, offering numerous benefits, such as a reduced area footprint and dynamically-adaptive computational precisio…
Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
Chao Fang, Man Shi, Robin Geens +3
The widely-used, weight-only quantized large language models (LLMs), which leverage low-bit integer (INT) weights and retain floating-point (FP) activations, reduce storage require…