9 citations · 15 across the 31 of their papers we have counts for
21 papers · 1 filter
Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents
Divyanshu Saxena, Rishikesh Maurya, Xiaoxuan Ou +7
The rapid adoption of AI agents across domains has made systematic evaluation crucial for ensuring their usefulness and successful production deployment. Evaluation of AI agents ty…
Simulating Environments with Reasoning Models for Agent Training
Yuetai Li, Huseyin A Inan, Xiang Yue +6
LLM agents excel in compact environments requiring deep reasoning but remain brittle when operating in broader, more complex contexts that demand robustness across diverse tools an…
SynthAgent: Adapting Web Agents with Synthetic Supervision
Zhaoyang Wang, Yiming Liang, Xuchao Zhang +9
Web agents struggle to adapt to new websites due to the scarcity of environment specific tasks and demonstrations. Recent works have explored synthetic data generation to address t…
Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems
Chenyu Zhao, Shenglin Zhang, Zeshun Huang +8
Large language models (LLMs) have shown growing potential in software engineering, yet few benchmarks evaluate their ability to repair software during migration across instruction…
Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth
Helia Hashemi, Victor Rühle, Saravan Rajmohan
Reasoning models have gained significant attention due to their strong performance, particularly when enhanced with retrieval augmentation. However, these models often incur high c…
LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automation
Dongge Han, Camille Couturier, Daniel Madrigal Diaz +3
We introduce LEGOMem, a modular procedural memory framework for multi-agent large language model (LLM) systems in workflow automation. LEGOMem decomposes past task trajectories int…
Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems
Fiza Husain, Anson Bastos, Anjaly Parayil +4
In this paper, we present DiRecGNN, an attention-enhanced entity recommendation framework for monitoring cloud services at Microsoft. We provide insights on the usefulness of this…
StepFly: Agentic Troubleshooting Guide Automation for Incident Diagnosis
Jiayi Mao, Liqun Li, Yanjie Gao +9
Effective incident management in large-scale IT systems relies on troubleshooting guides (TSGs), but their manual execution is slow and error-prone. While recent advances in LLMs o…
ACON: Optimizing Context Compression for Long-horizon LLM Agents
Minki Kang, Wei-Ning Chen, Dongge Han +5
Large language models (LLMs) are increasingly deployed as agents in dynamic real-world environments, where success depends on maintaining precise records of actions and observation…
OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows
Weixuan Wang, Dongge Han, Daniel Madrigal Diaz +3
Autonomous agents powered by large language models (LLMs) are increasingly deployed in real-world applications requiring complex, long-horizon workflows. However, existing benchmar…
Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
Dongge Han, Menglin Xia, Daniel Madrigal Diaz +7
Small language models (SLMs) offer promising and efficient alternatives to large language models (LLMs). However, SLMs' limited capacity restricts their reasoning capabilities and…
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang +12
The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs…
Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
Camille Couturier, Spyros Mastorakis, Haiying Shen +2
Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing leng…
Text2Grad: Reinforcement Learning from Natural Language Feedback
Hanyang Wang, Lu Wang, Chaoyun Zhang +5
Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow and opaque learning. Recent wor…
Exploring How LLMs Capture and Represent Domain-Specific Knowledge
Mirian Hipolito Garcia, Camille Couturier, Daniel Madrigal Diaz +5
We study whether Large Language Models (LLMs) inherently capture domain-specific nuances in natural language. Our experiments probe the domain sensitivity of LLMs by examining thei…
eARCO: Efficient Automated Root Cause Analysis with Prompt Optimization
Drishti Goel, Raghav Magazine, Supriyo Ghosh +5
Root cause analysis (RCA) for incidents in large-scale cloud systems is a complex, knowledge-intensive task that often requires significant manual effort from on-call engineers (OC…
AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Taneesh Gupta, Rahul Madhavan, Xuchao Zhang +2
Multi-preference optimization enriches language-model alignment beyond pairwise preferences by contrasting entire sets of helpful and undesired responses, thereby enabling richer t…
SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan +9
Global cloud service providers handle inference workloads for Large Language Models (LLMs) that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks…
Minerva: A Programmable Memory Test Benchmark for Language Models
Menglin Xia, Victor Ruehle, Saravan Rajmohan +1
How effectively can LLM-based AI assistants utilize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from seve…
DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
Linghao Zhang, Junhao Wang, Shilin He +13
Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and…
Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms
Benjamin Reidys, Pantea Zardoshti, Íñigo Goiri +16
Cloud platforms remain underutilized despite multiple proposals to improve their utilization (e.g., disaggregation, harvesting, and oversubscription). Our characterization of the r…