8 papers
Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off
Yuan Lyu, Bharath Irukulapati, Jaya Prakash Champati
Speculative decoding (SD) accelerates LLM inference by - times when the draft and target models are co-located. This has motivated a distributed variant (DSD) that places t…
Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference
Ziming Dong, Hardik Sharma, Evan O'Toole +2
Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offe…
Inference Offloading for Cost-Sensitive Binary Classification at the Edge
Vishnu Narayanan Moothedath, Umang Agarwal, Umeshraja N +3
We focus on a binary classification problem in an edge intelligence system where false negatives are more costly than false positives. The system has a compact, locally deployed mo…
Low-Regret and Low-Complexity Learning for Hierarchical Inference
Sameep Chattopadhyay, Vinay Sutar, Jaya Prakash Champati +1
This work focuses on Hierarchical Inference (HI) in edge intelligence systems, where a compact Local-ML model on an end-device works in conjunction with a high-accuracy Remote-ML m…
Error Bounds for the Network Scale-Up Method
Sergio DÃaz-Aranda, Juan Marcos RamÃrez, Mohit Daga +4
Epidemiologists and social scientists have used the Network Scale-Up Method (NSUM) for over thirty years to estimate the size of a hidden sub-population within a social network. Th…
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
Adarsh Prasad Behera, Jaya Prakash Champati, Roberto Morabito +2
Recent progress in Language Models (LMs) has dramatically advanced the field of natural language processing (NLP), excelling at tasks like text generation, summarization, and quest…