1 citations · 1 across the 2 of their papers we have counts for
7 papers
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
Jiaxuan Chen, Jianshu She, Ye Yuan +5
LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a h…
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Zonghang Li, Tao Li, Wenjiao Feng +8
On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome th…
Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification
Hong Huang, Decheng Wu, Qiangqiang Hu +5
The deployment of Large Language Models (LLMs) on resource-constrained edge devices is increasingly hindered by prohibitive memory and computational requirements. While ternary qua…
Extreme Model Compression with Structured Sparsity at Low Precision
Dan Liu, Nikita Dvornik, Xue Liu
Deep neural networks (DNNs) are used in many applications, but their large size and high computational cost make them hard to run on devices with limited resources. Two widely used…
FLASH Viterbi: Fast and Adaptive Viterbi Decoding for Modern Data Systems
Ziheng Deng, Xue Liu, Jiantong Jiang +3
The Viterbi algorithm is a key operator for structured sequence inference in modern data systems, with applications in trajectory analysis, online recommendation, and speech recogn…
Tequila: Trapping-free Ternary Quantization for Large Language Models
Hong Huang, Decheng Wu, Rui Cen +7
Quantization techniques are essential for the deployment of Large Language Models (LLMs) on edge devices. However, prevailing methods often rely on mixed-precision multiplication t…