2 citations · 5 across the 7 of their papers we have counts for
9 papers
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Soham Dan, Jeff Da, Sanyam Bhutani +8
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and in…
Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
Weijun Luo, Kelvin Luu, Xinyi Liu +6
Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes constitute unearned passes; their p…
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refac…
Improving Robustness of Tabular Retrieval via Representational Stability
Kushal Raj Bhandari, Adarsh Singh, Jianxi Gao +2
Transformer-based table retrieval systems flatten structured tables into token sequences, making retrieval sensitive to the choice of serialization even when table semantics remain…
Few-shot Policy (de)composition in Conversational Question Answering
Kyle Erwin, Guy Axelrod, Maria Chang +8
The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of…
On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems
Md Ibrahim Ibne Alam, Parikshit Ram, Soham Dan +2
Large Language Models (LLMs) have been observed to perform well on a wide range of downstream tasks when fine-tuned on domain-specific data. However, such data may not be readily a…
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury +7
There is a growing need for Large Language Models (LLMs) to effectively use tools and external Application Programming Interfaces (APIs) to plan and complete tasks. As such, there…
Formally Specifying the High-Level Behavior of LLM-Based Agents
Maxwell Crouse, Ibrahim Abdelaziz, Ramon Astudillo +7
Autonomous, goal-driven agents powered by LLMs have recently emerged as promising tools for solving challenging problems without the need for task-specific finetuned models that ca…
Compositional Program Generation for Few-Shot Systematic Generalization
Tim Klinger, Luke Liu, Soham Dan +3
Compositional generalization is a key ability of humans that enables us to learn new concepts from only a handful examples. Neural machine learning models, including the now ubiqui…