Niyama : Breaking the Silos of LLM Inference Serving

Kanishk Goel, Jayashree Mohan, Nipun Kwatra +2

The widespread adoption of Large Language Models (LLMs) has enabled diverse applications with very different latency requirements. Existing LLM serving frameworks rely on siloed in…