5 papers
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
Sunjoo Whang, Jungjun Oh, Minsung Kim +5
Long inputs and extended generation increase the storage and access costs of the key-value (KV) cache. Low-bit quantization reduces storage and memory traffic, while query-channel…
CircuitsDNA: Discovering Unconventional Multi-Accuracy Arithmetic Circuits via Evolutionary Synthesis
Ruichen Qi, Junyi Luo, Xinting Jiang +5
Emerging edge AI workloads increasingly require arithmetic units that can trade computational accuracy for efficiency on demand. However, existing approximate arithmetic circuits a…
Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
Junyi Luo, Xinting Jiang, Tai-Hao Wen +9
Microscaling (MX) is now the standard for low-bit large language model (LLM) inference. Its 4-bit form MXFP4 still loses substantial accuracy, because existing MX formats fix eithe…
LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
Xinting Jiang, Junyi Luo, Ruichen Qi +4
Sub-billion-parameter Transformer language models are increasingly deployed on edge devices, where the privacy, latency, and operating-cost advantages of on-device inference are co…
ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters
Shiwei Liu, Guanchen Tao, Yifei Zou +7
The self-attention mechanism distinguishes transformer-based large language models (LLMs) apart from convolutional and recurrent neural networks. Despite the performance improvemen…