DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving

Fengze Yu, Leshu Li, Brad McDanel +1

Large language model (LLM) inference often suffers from high decoding latency and limited scalability across heterogeneous edge-cloud environments. Existing speculative decoding (S…