1 citations · 6 across the 34 of their papers we have counts for
36 papers
On-Policy Attention Linearization
Arian Raje, Anupam Nayak, Anthony Fei +3
Hybrid transformer architectures that replace most softmax attention layers with linear attention offer transformer-level quality at a fraction of the memory cost. Rather than pret…
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
Chun-Ting Chen, Dongmin Han, Hangyeol Mun +6
Block Quantization (BQ) enables efficient LLM inference by quantizing both weights and activations, but its design space remains underexplored. Through hardware-accuracy design spa…
Compute Where it Counts: Self Optimizing Language Models
Yash Akhauri, Mohamed S. Abdelfattah
Efficient LLM inference research has largely focused on reducing the cost of each decoding step (e.g., using quantization, pruning, or sparse attention), typically applying a unifo…
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suf…
Faster LLM Inference via Sequential Monte Carlo
Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4
Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sam…
UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
Hung-Yueh Chiang, Chi-Chih Chang, Yu-Chen Lu +4
Deploying large language models (LLMs) on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availab…
Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
Yilong Zhao, Jiaming Tang, Kan Zhu +10
Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation…
P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats
Yuzong Chen, Chao Fang, Xilai Dai +4
The substantial memory bandwidth and computational demands of large language models (LLMs) present critical challenges for efficient inference. To tackle this, the literature has e…
Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
Pei-Shuo Wang, Jian-Jia Chen, Chun-Che Yang +4
The immense model sizes of large language models (LLMs) challenge deployment on memory-limited consumer GPUs. Although model compression and parameter offloading are common strateg…
Regression Language Models for Code
Yash Akhauri, Xingyou Song, Arissa Wongpanich +2
We study code-to-metric regression: predicting numeric outcomes of code executions, a challenging task due to the open-ended nature of programming languages. While prior methods ha…
Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference
Chun-Ting Chen, HanGyeol Mun, Jian Meng +2
Edge inference for large language models (LLM) offers secure, low-latency, and cost-effective inference solutions. We emphasize that an edge accelerator should achieve high area ef…
Performance Prediction for Large Systems via Text-to-Text Regression
Yash Akhauri, Bryan Lewandowski, Cheng-Hsi Lin +7
In many industries, predicting metric outcomes of large systems is a fundamental problem, driven largely by traditional tabular regression. However, such methods struggle on comple…
Semantic Compression of 3D Objects for Open and Collaborative Virtual Worlds
Jordan Dotzel, Tony Montes, Mohamed S. Abdelfattah +1
Traditional methods for 3D object compression operate only on structural information within the object vertices, polygons, and textures. These methods are effective at compression…
FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
Zhanqiu Hu, Jian Meng, Yash Akhauri +4
Diffusion language models offer parallel token generation and inherent bidirectionality, promising more efficient and powerful sequence modeling compared to autoregressive approach…
SplitReason: Learning To Offload Reasoning
Yash Akhauri, Anthony Fei, Chi-Chih Chang +3
Reasoning in large language models (LLMs) tends to produce substantially longer token generation sequences than simpler language modeling tasks. This extended generation length ref…
Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin +3
State Space Models (SSMs) are emerging as a compelling alternative to Transformers because of their consistent memory usage and high performance. Despite this, scaling up SSMs on c…
xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin +8
Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache…
TokenButler: Token Importance is Predictable
Yash Akhauri, Ahmed F AbouElhamayed, Yifei Gao +4
Large Language Models (LLMs) rely on the Key-Value (KV) Cache to store token history, enabling efficient decoding of tokens. As the KV-Cache grows, it becomes a major memory and co…
SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs
Ahmed F. AbouElhamayed, Jordan Dotzel, Yash Akhauri +6
Large language models have high compute, latency, and memory requirements. While specialized accelerators such as GPUs and TPUs typically run these workloads, CPUs are more widely…
RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping
Yuzong Chen, Xilai Dai, Jake Hyun +6
The recently introduced NVFP4 format demonstrates remarkable performance and memory benefits for quantized large language model (LLM) inference. However, we observe two types of re…
NITRO: LLM Inference on Intel Laptop NPUs
Anthony Fei, Mohamed S. Abdelfattah
Large Language Models (LLMs) have become essential tools in natural language processing, finding large usage in chatbots such as ChatGPT and Gemini, and are a central area of resea…
Attamba: Attending To Multi-Token States
Yash Akhauri, Safeen Huda, Mohamed S. Abdelfattah
When predicting the next token in a sequence, vanilla transformers compute attention over all previous tokens, resulting in quadratic scaling of compute with sequence length. State…
BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration
Yuzong Chen, Ahmed F. AbouElhamayed, Xilai Dai +4
Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders thei…
BBS: Bi-directional Bit-level Sparsity for Deep Learning Acceleration
Yuzong Chen, Jian Meng, Jae-sun Seo +1
Bit-level sparsity methods skip ineffectual zero-bit operations and are typically applicable within bit-serial deep learning accelerators. This type of sparsity at the bit-level is…