3 citations · 12 across the 19 of their papers we have counts for
21 papers
Understanding KV Residency in Production-Scale Agentic LLM Workloads
Mayank Arya, Agrim Bari, Anjaly Parayil +4
Agentic large language model (LLM) applications turn one user request into a sequence of model and tool calls over a growing context. Retaining this context's key-value (KV) state…
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj +2
Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tail…
XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farms
Tella Rajashekhar Reddy, Atharva Deshmukh, Liangcheng Yu +7
AI power demand is growing at an unprecedented rate while power grids are often ailing and struggle to keep up. Grid expansion comes with high capital expenditure and long-distance…
A Holistic Framework for Automated Configuration Recommendation for Cloud Service Monitoring
Anson Bastos, Shreeya Venneti, Anjaly Parayil +3
Reliability of large-scale cloud services is critical for user satisfaction and business continuity. Despite significant investments in reliability engineering, production incident…
Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
Anish Biswas, Kanishk Goel, Srivarshinee S +5
Agentic applications are LLMs that iteratively invoke external tools to accomplish complex tasks. Such tool-based agents are rapidly becoming the dominant paradigm for deploying la…
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
Shashwat Jaiswal, Shrikara Arun, Anjaly Parayil +8
Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In prod…
BeLLMan: Controlling LLM Congestion
Tella Rajashekhar Reddy, Atharva Deshmukh, Karan Tandon +3
Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferenc…
Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems
Fiza Husain, Anson Bastos, Anjaly Parayil +4
In this paper, we present DiRecGNN, an attention-enhanced entity recommendation framework for monitoring cloud services at Microsoft. We provide insights on the usefulness of this…
AI Greenferencing: Routing AI Inferencing to Green Modular Data Centers with Heron
Tella Rajashekhar Reddy, Palak, Rohan Gandhi +8
AI power demand is growing unprecedentedly thanks to the high power density of AI compute and the emerging inferencing workload. On the supply side, abundant wind power is waiting…
SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan +9
Global cloud service providers handle inference workloads for Large Language Models (LLMs) that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks…
Ensuring Fair LLM Serving Amid Diverse Applications
Redwan Ibne Seraj Khan, Kunal Jain, Haiying Shen +12
In a multi-tenant large language model (LLM) serving platform hosting diverse applications, some users may submit an excessive number of requests, causing the service to become una…
Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTC
Aditya Soni, Mayukh Das, Anjaly Parayil +8
The difficulty of exploring and training online on real production systems limits the scope of real-time online data/feedback-driven decision making. The most feasible approach is…
Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing
Kunal Jain, Anjaly Parayil, Ankur Mallick +10
Large Language Model (LLM) workloads have distinct prefill and decode phases with different compute and memory requirements which should ideally be accounted for when scheduling in…
Towards Cloud Efficiency with Large-scale Workload Characterization
Anjaly Parayil, Jue Zhang, Xiaoting Qin +4
Cloud providers introduce features (e.g., Spot VMs, Harvest VMs, and Burstable VMs) and optimizations (e.g., oversubscription, auto-scaling, power harvesting, and overclocking) to…
Workload Intelligence: Punching Holes Through the Cloud Abstraction
Lexiang Huang, Anjaly Parayil, Jue Zhang +13
Today, cloud workloads are essentially opaque to the cloud platform. Typically, the only information the platform receives is the virtual machine (VM) type and possibly a decoratio…
X-lifecycle Learning for Cloud Incident Management using LLMs
Drishti Goel, Fiza Husain, Aditya Singh +5
Incident management for large cloud services is a complex and tedious process and requires significant amount of manual efforts from on-call engineers (OCEs). OCEs typically levera…
Intelligent Monitoring Framework for Cloud Services: A Data-Driven Approach
Pooja Srinivas, Fiza Husain, Anjaly Parayil +3
Cloud service owners need to continuously monitor their services to ensure high availability and reliability. Gaps in monitoring can lead to delay in incident detection and signifi…
On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spaces
Amrit Singh Bedi, Souradip Chakraborty, Anjaly Parayil +3
We focus on parameterized policy search for reinforcement learning over continuous action spaces. Typically, one assumes the score function associated with a policy is bounded, whi…
On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control
Amrit Singh Bedi, Anjaly Parayil, Junyu Zhang +2
Reinforcement learning is a framework for interactive decision-making with incentives sequentially revealed across time without a system dynamics model. Due to its scaling to conti…
A Decentralized Approach to Bayesian Learning
Anjaly Parayil, He Bai, Jemin George +1
Motivated by decentralized approaches to machine learning, we propose a collaborative Bayesian learning algorithm taking the form of decentralized Langevin dynamics in a non-convex…
Multi-Agent Coordination for Distributed Transmit Beamforming
Jemin George, Anjaly Parayil, He Bai
This paper presents the formulation and analysis of a two time-scale optimization algorithm for multi-agent coordination for the purpose of distributed beamforming. Each agent is a…