9 papers
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6
While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck whe…
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on…
Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression
Hengyi Zhu, Zhendong Mi, Grace Li Zhang +1
Large language models (LLMs) have rapidly grown in scale, creating substantial memory and computational costs that hinder efficient deployment. Singular value decomposition (SVD) h…
Late Breaking Results: Conversion of Neural Networks into Logic Flows for Edge Computing
Daniel Stein, Shaoyi Huang, Rolf Drechsler +2
Neural networks have been successfully applied in various resource-constrained edge devices, where usually central processing units (CPUs) instead of graphics processing units exis…
Layer-wise Weight Selection for Power-Efficient Neural Network Acceleration
Jiaxun Fang, Grace Li Zhang, Shaoyi Huang
Systolic array accelerators execute CNNs with energy dominated by the switching activity of multiply accumulate (MAC) units. Although prior work exploits weight dependent MAC power…
LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Search
Hengyi Zhu, Grace Li Zhang, Shaoyi Huang
Hardware-Aware Neural Architecture Search (HW-NAS) requires joint optimization of accuracy and latency under device constraints. Traditional supernet-based methods require multiple…