91 citations · 123 across the 24 of their papers we have counts for
28 papers
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3
Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up…
Online Monitoring and Corrective Steering of Programming Agents
Shuyang Liu, Saman Dehghan, Ji Young Kim +3
Fixing GitHub issues in large-scale projects is a long-horizon task, especially when a fix requires changes across multiple locations or the issue description lacks the information…
PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization
Ryan Deng, Yuanzhe Liu, Bastian Lipka +4
Large language model (LLM) agents now perform well on correctness-oriented repository-level tasks, including SWE-Bench issue resolution and feature implementation in real codebases…
EvoOtter: Evolutionary Reproduction Test Generator
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descri…
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First…
Reproduction Test Generation for Java SWE Issues
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Given an issue on a software repository, a reproduction test confirms its presence in the code before it gets fixed and its absence after. Reproduction tests provide crucial execut…
From Plan to Action: How Well Do Agents Follow the Plan?
Shuyang Liu, Saman Dehghan, Jatin Ganhotra +2
Agents are commonly instructed to follow a task-specific plan for guidance. However, it is unknown to what extent agents actually follow instructed plans. Without such an analysis,…
Resolving Java Code Repository Issues with iSWE Agent
Jatin Ganhotra, Sami Serhan, Antonio Abu Nassar +3
Resolving issues on code repositories is an important part of software engineering. Various recent systems automatically resolve issues using large language models and agents, ofte…
Process-Centric Analysis of Agentic Software Systems
Shuyang Liu, Yang Chen, Rahul Krishna +3
Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their ex…
Investigating Test Overfitting on SWE-bench
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests b…
When Agents go Astray: Course-Correcting SWE Agents with PRMs
Shubham Gandhi, Jason Tsay, Jatin Ganhotra +2
Large Language Model (LLM) agents are increasingly deployed for complex, multi-step software engineering (SWE) tasks. However, their trajectories often contain costly inefficiencie…
Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, s…
Otter: Generating Tests from Issues to Validate SWE Patches
Toufique Ahmed, Jatin Ganhotra, Rangeet Pan +3
While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch…
Structured Chain-of-Thought Prompting for Few-Shot Generation of Content-Grounded QA Conversations
Md Arafat Sultan, Jatin Ganhotra, Ramón Fernandez Astudillo
We introduce a structured chain-of-thought (SCoT) prompting approach to generating content-grounded multi-turn question-answer conversations using a pre-trained large language mode…
Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variations
Ankita Gupta, Chulaka Gunasekara, Hui Wan +3
Dialogue summarization task involves summarizing long conversations while preserving the most salient information. Real-life dialogues often involve naturally occurring variations…
Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding
Vishal Sunder, Samuel Thomas, Hong-Kwang J. Kuo +3
Dialog history plays an important role in spoken language understanding (SLU) performance in a dialog system. For end-to-end (E2E) SLU, previous work has used dialog history in tex…
Integrating Dialog History into End-to-End Spoken Language Understanding Systems
Jatin Ganhotra, Samuel Thomas, Hong-Kwang J. Kuo +4
End-to-end spoken language understanding (SLU) systems that process human-human or human-computer interactions are often context independent and process each turn of a conversation…
Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
Hanjie Chen, Song Feng, Jatin Ganhotra +4
Explaining neural network models is important for increasing their trustworthiness in real-world applications. Most existing methods generate post-hoc explanations for neural netwo…
Does Dialog Length matter for Next Response Selection task? An Empirical Study
Jatin Ganhotra, Sachindra Joshi
In the last few years, the release of BERT, a multilingual transformer based model, has taken the NLP community by storm. BERT-based models have achieved state-of-the-art results o…
Conversational Document Prediction to Assist Customer Care Agents
Jatin Ganhotra, Haggai Roitman, Doron Cohen +6
A frequent pattern in customer care conversations is the agents responding with appropriate webpage URLs that address users' needs. We study the task of predicting the documents th…
Effects of Naturalistic Variation in Goal-Oriented Dialog
Jatin Ganhotra, Robert Moore, Sachindra Joshi +1
Existing benchmarks used to evaluate the performance of end-to-end neural dialog systems lack a key component: natural variation present in human conversations. Most datasets are c…
Learning End-to-End Goal-Oriented Dialog with Maximal User Task Success and Minimal Human Agent Use
Janarthanan Rajendran, Jatin Ganhotra, Lazaros Polymenakos
Neural end-to-end goal-oriented dialog systems showed promise to reduce the workload of human agents for customer service, as well as reduce wait time for users. However, their ina…
Knowledge-incorporating ESIM models for Response Selection in Retrieval-based Dialog Systems
Jatin Ganhotra, Siva Sankalp Patel, Kshitij Fadnis
Goal-oriented dialog systems, which can be trained end-to-end without manually encoding domain-specific features, show tremendous promise in the customer support use-case e.g. flig…
Quantized-Dialog Language Model for Goal-Oriented Conversational Systems
R. Chulaka Gunasekara, David Nahamoo, Lazaros C. Polymenakos +2
We propose a novel methodology to address dialog learning in the context of goal-oriented conversational systems. The key idea is to quantize the dialog space into clusters and cre…