10 papers
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6
While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck whe…
FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference
Yu-Chen Lu, Chong-Yan Chen, Chi-Chih Chang +2
Although large language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression…
Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
Pei-Shuo Wang, Jian-Jia Chen, Chun-Che Yang +4
The immense model sizes of large language models (LLMs) challenge deployment on memory-limited consumer GPUs. Although model compression and parameter offloading are common strateg…
Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin +3
State Space Models (SSMs) are emerging as a compelling alternative to Transformers because of their consistent memory usage and high performance. Despite this, scaling up SSMs on c…
Systolic Sparse Tensor Slices: FPGA Building Blocks for Sparse and Dense AI Acceleration
Endri Taka, Ning-Chi Huang, Chi-Chih Chang +3
FPGA architectures have recently been enhanced to meet the substantial computational demands of modern deep neural networks (DNNs). To this end, both FPGA vendors and academic rese…
V"Mean"ba: Visual State Space Models only need 1 hidden dimension
Tien-Yu Chi, Hung-Yueh Chiang, Chi-Chih Chang +2
Vision transformers dominate image processing tasks due to their superior performance. However, the quadratic complexity of self-attention limits the scalability of these systems a…
Quamba: A Post-Training Quantization Recipe for Selective State Space Models
Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin +2
State Space Models (SSMs) have emerged as an appealing alternative to Transformers for large language models, achieving state-of-the-art accuracy with constant memory complexity wh…
ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration
Ning-Chi Huang, Chi-Chih Chang, Wei-Cheng Lin +3
sparsity is an emerging model compression method supported by more and more accelerators to speed up sparse matrix multiplication in deep neural networks. Most existing $N{…
Palu: Compressing KV-Cache with Low-Rank Projection
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin +7
Post-training KV-Cache compression methods typically either sample a subset of effectual tokens or quantize the data into lower numerical bit width. However, these methods cannot e…
FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer
Chi-Chih Chang, Yuan-Yao Sung, Shixing Yu +3
Vision Transformers (ViT) have recently demonstrated success across a myriad of computer vision tasks. However, their elevated computational demands pose significant challenges for…