46 citations · 104 across the 35 of their papers we have counts for
40 papers
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3
Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up…
One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models
Siqi Yang, Qianlan Yang, Yu-Xiong Wang +2
Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluat…
Online Monitoring and Corrective Steering of Programming Agents
Shuyang Liu, Saman Dehghan, Ji Young Kim +3
Fixing GitHub issues in large-scale projects is a long-horizon task, especially when a fix requires changes across multiple locations or the issue description lacks the information…
PPDL: LLM-Based Flows as Probabilistic Programs
Louis Mandel, Guillaume Baudart, Mandana Vaziri +1
Building reliable applications that leverage large language models (LLMs) remains a significant challenge. While LLMs offer impressive capabilities across diverse tasks, their outp…
EvoOtter: Evolutionary Reproduction Test Generator
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descri…
Reproduction Test Generation for Java SWE Issues
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Given an issue on a software repository, a reproduction test confirms its presence in the code before it gets fixed and its absence after. Reproduction tests provide crucial execut…
From Plan to Action: How Well Do Agents Follow the Plan?
Shuyang Liu, Saman Dehghan, Jatin Ganhotra +2
Agents are commonly instructed to follow a task-specific plan for guidance. However, it is unknown to what extent agents actually follow instructed plans. Without such an analysis,…
Resolving Java Code Repository Issues with iSWE Agent
Jatin Ganhotra, Sami Serhan, Antonio Abu Nassar +3
Resolving issues on code repositories is an important part of software engineering. Various recent systems automatically resolve issues using large language models and agents, ofte…
Investigating Test Overfitting on SWE-bench
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests b…
Can Old Tests Do New Tricks for Resolving SWE Issues?
Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand +1
Test suites in real-world projects are often large and achieve high code coverage, yet they remain insufficient for detecting all bugs. The abundance of unresolved issues in open-s…
Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, s…
Representing Prompting Patterns with PDL: Compliance Agent Case Study
Mandana Vaziri, Louis Mandel, Yuji Watanabe +3
Prompt engineering for LLMs remains complex, with existing frameworks either hiding complexity behind restrictive APIs or providing inflexible canned patterns that resist customiza…
Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning
Kush Jain, Kiran Kate, Jason Tsay +2
Examples in web API specifications can be essential for API testing, API understanding, and even building chat-bots for APIs. Unfortunately, most API specifications lack human-writ…
AutoPDL: Automatic Prompt Optimization for LLM Agents
Claudio Spiess, Mandana Vaziri, Louis Mandel +1
The performance of large language models (LLMs) depends on how they are prompted, with choices spanning both the high-level prompting pattern (e.g., Zero-Shot, CoT, ReAct, ReWOO) a…
Otter: Generating Tests from Issues to Validate SWE Patches
Toufique Ahmed, Jatin Ganhotra, Rangeet Pan +3
While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch…
TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?
Toufique Ahmed, Martin Hirzel, Rangeet Pan +2
Test-driven development (TDD) is the practice of writing tests first and coding later, and the proponents of TDD expound its numerous benefits. For instance, given an issue on a so…
Transforming the Hybrid Cloud for Emerging AI Workloads
Deming Chen, Alaa Youssef, Ruchi Pendse +42
This white paper, developed through close collaboration between IBM Research and UIUC researchers within the IIDAI Institute, envisions transforming hybrid cloud systems to meet th…
PDL: A Declarative Prompt Programming Language
Mandana Vaziri, Louis Mandel, Claudio Spiess +1
Large language models (LLMs) have taken the world by storm by making many previously difficult uses of AI feasible. LLMs are controlled via highly expressive textual prompts and re…
PoTo: A Hybrid Andersen's Points-to Analysis for Python
Ingkarat Rak-amnouykit, Ana Milanova, Guillaume Baudart +2
As Python is increasingly being adopted for large and complex programs, the importance of static analysis for Python (such as type inference) grows. Unfortunately, static analysis…
Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study
Razan Baltaji, Saurabh Pujar, Louis Mandel +3
Large language models (LLMs) have achieved state-of-the-art performance in various software engineering tasks, including error detection, clone detection, and code translation, pri…
A Suite of Fairness Datasets for Tabular Classification
Martin Hirzel, Michael Feffer
There have been many papers with algorithms for improving fairness of machine-learning classifiers for tabular data. Unfortunately, most use only very few datasets for their experi…
Out-of-Order Sliding-Window Aggregation with Efficient Bulk Evictions and Insertions (Extended Version)
Kanat Tangwongsan, Martin Hirzel, Scott Schneider
Sliding-window aggregation is a foundational stream processing primitive that efficiently summarizes recent data. The state-of-the-art algorithms for sliding-window aggregation are…
AI for Low-Code for AI
Nikitha Rao, Jason Tsay, Kiran Kate +2
Low-code programming allows citizen developers to create programs with minimal coding effort, typically via visual (e.g. drag-and-drop) interfaces. In parallel, recent AI-powered t…
Navigating Ensemble Configurations for Algorithmic Fairness
Michael Feffer, Martin Hirzel, Samuel C. Hoffman +3
Bias mitigators can improve algorithmic fairness in machine learning models, but their effect on fairness is often not stable across data splits. A popular approach to train more s…