1 citations · 3 across the 9 of their papers we have counts for
14 papers
Factorization-Guided Training and Inference for Diffusion-Drafted Speculative Decoding
Jiakun Fan, Min Xiao, Qiang Wang +5
Speculative decoding accelerates large language model inference through parallel verification of tokens proposed by a drafter. Diffusion drafters further parallelize proposal gener…
A Framework for Accelerating Transformer Inference on RISC-V for Edge AI
Ajay Kumar M, Vishnu PS, Yike Li +5
This work presents a framework for accelerating transformer-based language models (LMs) on resource-constrained IoT devices. The framework targets compact LMs: BERT-Tiny (B-Ty), Mo…
ZTA-Q: an Open-source RISC-V Platform for Accurate Quantized CNN Inference
Yike Li, Ajay Kumar M, Vishnu PS +4
Low-precision inference is widely adopted in edge AI to reduce computational cost and memory footprint. However, existing open-source accelerator platforms provide limited end-to-e…
FACT: Compositional Kernel Synthesis with a Three-Stage Agentic Workflow
Sina Heidari, Dimitrios S. Nikolopoulos
Deep learning compilers and vendor libraries deliver strong baseline performance but their performance is bounded by finite, engineer-curated catalogs. When these omit needed optim…
LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference
Katelyn Crumpacker, Dimitrios Nikolopoulos
Large Language Models (LLMs) have become an integral part of many real-world workflows. However, LLMs consume a lot of energy, which becomes a large concern in the scale of the dem…
ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
Xiangchen Li, Saeid Ghafouri, Jiakun Fan +3
Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While…
Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving
Jiakun Fan, Yanglin Zhang, Xiangchen Li +1
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to Autoregressive Models (ARMs), utilizing parallel decoding to overcome sequential bottlenecks. How…
WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
Xiangchen Li, Jiakun Fan, Qingyuan Wang +7
As Large Language Models (LLMs) become increasingly accessible to end users, an ever-growing number of inference requests are initiated from edge devices and computed on centralize…
MARVEL: An End-to-End Framework for Generating Model-Class Aware Custom RISC-V Extensions for Lightweight AI
Ajay Kumar M, Cian O'Mahoney, Pedro Kreutz Werle +5
Deploying deep neural networks (DNNs) on resource-constrained IoT devices remains a challenging problem, often requiring hardware modifications tailored to individual AI models. Ex…
Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices
Saeid Ghafouri, Mohsen Fayyaz, Xiangchen Li +4
Real-time multi-label video classification on embedded devices is constrained by limited compute and energy budgets. Yet, video streams exhibit structural properties such as label…
QPART: Adaptive Model Quantization and Dynamic Workload Balancing for Accuracy-aware Edge Inference
Xiangchen Li, Saeid Ghafouri, Bo Ji +3
As machine learning inferences increasingly move to edge devices, adapting to diverse computational capabilities, hardware, and memory constraints becomes more critical. Instead of…
SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
Xiangchen Li, Dimitrios Spatharakis, Saeid Ghafouri +5
The growing gap between the increasing complexity of large language models (LLMs) and the limited computational budgets of edge devices poses a key challenge for efficient on-devic…
APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
Jiakun Fan, Yanglin Zhang, Xiangchen Li +1
Deploying large language models (LLMs) for online inference is often constrained by limited GPU memory, particularly due to the growing KV cache during auto-regressive decoding. Hy…
ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments
Munkyu Lee, Sihoon Seong, Minki Kang +5
In cloud environments, GPU-based deep neural network (DNN) inference servers are required to meet the Service Level Objective (SLO) latency for each workload under a specified requ…