Publications (32)
EcoServe: Designing Carbon-Aware AI Inference Systems
Yueying Li, Zhanqiu Hu, Esha Choukse +3
The rapid increase in LLM ubiquity and scale levies unprecedented demands on computing infrastructure. These demands not only incur large compute and memory resources but also sign…
Nanvix: A Multikernel OS Design for High-Density Serverless Deployments
Carlos Segarra, Pedro Henrique Penna, Enrique Saurez +4
Serverless providers strive for high resource utilization by optimizing deployment density: how many applications can be deployed per host server. However, achieving high deploymen…
WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
Madhav Kanda, Sharad Agarwal, Rodrigo Fonseca +2
Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise process orc…
PACE-LM: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysis
Dylan Zhang, Xuchao Zhang, Chetan Bansal +3
Major cloud providers have employed advanced AI-based solutions like large language models to aid humans in identifying the root causes of cloud incidents. Despite the growing prev…
Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
Mohammad Shahrad, Rodrigo Fonseca, Ãñigo Goiri +7
Function as a Service (FaaS) has been gaining popularity as a way to deploy computations to serverless backends in the cloud. This paradigm shifts the complexity of allocating and…
Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications
Francisco Romero, Gohar Irfan Chaudhry, Ãñigo Goiri +6
Function-as-a-Service (FaaS) has become an increasingly popular way for users to deploy their applications without the burden of managing the underlying infrastructure. However, ex…
ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
Haoran Qiu, Anish Biswas, Zihan Zhao +9
Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text. However, efficiently serving LMMs in production environme…
Workload Intelligence: Punching Holes Through the Cloud Abstraction
Lexiang Huang, Anjaly Parayil, Jue Zhang +13
Today, cloud workloads are essentially opaque to the cloud platform. Typically, the only information the platform receives is the virtual machine (VM) type and possibly a decoratio…
Towards Resource-Efficient Compound AI Systems
Gohar Irfan Chaudhry, Esha Choukse, Ãñigo Goiri +3
Compound AI Systems, integrating multiple interacting components like models, retrievers, and external tools, have emerged as essential for addressing complex AI tasks. However, cu…
Junctiond: Extending FaaS Runtimes with Kernel-Bypass
Enrique Saurez, Joshua Fried, Gohar Irfan Chaudhry +5
This report explores the use of kernel-bypass networking in FaaS runtimes and demonstrates how using Junction, a novel kernel-bypass system, as the backend for executing components…
SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks
Linnan Wang, Wei Wu, Junyu Zhang +4
The performance and efficiency of distributed training of Deep Neural Networks highly depend on the performance of gradient averaging among all participating nodes, which is bounde…
A Performance Analyzer for a Public Cloud's ML-Augmented VM Allocator
Roozbeh Bostandoost, Pooria Namyar, Siva Kesava Reddy Kakarla +8
Cloud operators increasingly deploy multiple ML models in their VM allocation pipelines. In such settings, individually benign predictions can shift and compound, severely degradin…
Sample-Efficient Neural Architecture Search by Learning Action Space
Linnan Wang, Saining Xie, Teng Li +2
Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing MCTS based NAS approaches often utilize manually design…
Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
Gohar Irfan Chaudhry, Esha Choukse, Haoran Qiu +4
Agentic workflows commonly coordinate multiple models and tools with complex control logic. They are quickly becoming the dominant paradigm for AI applications. However, serving th…
Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search
Linnan Wang, Rodrigo Fonseca, Yuandong Tian
High dimensional black-box optimization has broad applications but remains a challenging problem to solve. Given a set of samples $\{\vx_i, y_i\}$, building a global model (like Ba…
Few-shot Neural Architecture Search
Yiyang Zhao, Linnan Wang, Yuandong Tian +2
Efficient evaluation of a network architecture drawn from a large search space remains a key challenge in Neural Architecture Search (NAS). Vanilla NAS evaluates each architecture…
Exploring LLM-based Agents for Root Cause Analysis
Devjeet Roy, Xuchao Zhang, Rashi Bhave +4
The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA…
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
Banruo Liu, Haoran Qiu, Ãñigo Goiri +3
AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots. We present the fi…
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Yueying Li, Yuanfan Chen, Jiayang Chen +6
LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ra…
Sherlock: Reliable and Efficient Agentic Workflow Execution
Yeonju Ro, Haoran Qiu, Ãñigo Goiri +6
With the increasing adoption of large language models (LLM), agentic workflows, which compose multiple LLM calls with tools, retrieval, and reasoning steps, are increasingly replac…
Compiling Stateful Network Properties for Runtime Verification
Tim Nelson, Nicholas DeMarinis, Timothy Adam Hoff +2
Networks are difficult to configure correctly, and tricky to debug. These problems are accentuated by temporal and stateful behavior. Static verification, while useful, is ineffect…
StreamWise: Serving Multi-Modal Generation in Real-Time at Scale
Haoran Qiu, Gohar Irfan Chaudhry, Chaojie Zhang +4
Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image…
With Great Freedom Comes Great Opportunity: Rethinking Resource Allocation for Serverless Functions
Muhammad Bilal, Marco Canini, Rodrigo Fonseca +1
Current serverless offerings give users a limited degree of flexibility for configuring the resources allocated to their function invocations by either coupling memory and CPU reso…
AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
Linnan Wang, Yiyang Zhao, Yuu Jinnai +2
Neural Architecture Search (NAS) has shown great success in automating the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires…
Enabling Multi-Dimensional Distributed Trace Comparison with Contrast
Vaastav Anand, Rodrigo Fonseca, Jonathan Mace +1
Diagnosis using distributed traces is fundamentally a comparative task: operators seek to understand how an anomalous execution differs from expected behavior, how a deployment cha…
Octopus: Enhancing CXL Memory Pods via Sparse Topology
Yuhong Zhong, Fiodar Kazhamiaka, Pantea Zardoshti +4
The Compute Express Link (CXL) interconnect enables compute "pods" that pool memory across servers to reduce cost and improve efficiency. These pods also facilitate pairwise commun…
Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving
Chengyi Nie, Rodrigo Fonseca, Zhenhua Liu
The demand for large language model (LLM) inference is gradually dominating the artificial intelligence workloads. Therefore, there is an urgent need for cost-efficient inference s…
TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
Jovan Stojkovic, Chaojie Zhang, Ãñigo Goiri +5
The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in cloud datacenters. Traditional techniques often are inadequate fo…
FITing-Tree: A Data-aware Index Structure
Alex Galakatos, Michael Markovitch, Carsten Binnig +2
Index structures are one of the most important tools that DBAs leverage to improve the performance of analytics and transactional workloads. However, building several indexes over…
Intent-based System Design and Operation
Vaastav Anand, Yichen Li, Alok Gautam Kumbhare +6
Cloud systems are the backbone of today's computing industry. Yet, these systems remain complicated to design, build, operate, and improve. All these tasks require significant manu…
Scanning the Internet for ROS: A View of Security in Robotics Research
Nicholas DeMarinis, Stefanie Tellex, Vasileios Kemerlis +2
Because robots can directly perceive and affect the physical world, security issues take on particular importance. In this paper, we describe the results of our work on scanning th…
Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
Linnan Wang, Yiyang Zhao, Yuu Jinnai +2
Neural Architecture Search (NAS) has shown great success in automating the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires…