492 citations · 1.5k across the 65 of their papers we have counts for
6 papers · 1 filter
RedunCut: Measurement-Driven Sampling and Accuracy Performance Modeling for Low-Cost Live Video Analytics
Gur-Eyal Sela, Kumar Krishna Agrawal, Bharathan Balaji +2
Live video analytics (LVA) runs continuously across massive camera fleets, but inference cost with modern vision models remains high. To address this, dynamic model size selection…
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Hanchen Li, Runyuan He, Qiuyang Mang +8
KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. Thi…
Towards Efficient and Practical GPU Multitasking in the Era of LLM
Jiarong Xing, Yifan Qiao, Simon Mo +5
GPU singletasking is becoming increasingly inefficient and unsustainable as hardware capabilities grow and workloads diversify. We are now at an inflection point where GPUs must em…
Sleep-time Compute: Beyond Inference Scaling at Test-time
Kevin Lin, Charlie Snell, Yu Wang +4
Scaling test-time compute has emerged as a key ingredient for enabling large language models (LLMs) to solve difficult problems, but comes with high latency and inference cost. We…
Jenga: Effective Memory Management for Serving LLM with Heterogeneity
Chen Zhang, Kuntai Du, Shu Liu +10
Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory…
BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
Alan Zhu, Parth Asawa, Jared Quincy Davis +5
As the demand for high-quality data in model training grows, researchers and developers are increasingly generating synthetic data to tune and train LLMs. However, current data gen…