4 papers
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refac…
Improving Robustness of Tabular Retrieval via Representational Stability
Kushal Raj Bhandari, Adarsh Singh, Jianxi Gao +2
Transformer-based table retrieval systems flatten structured tables into token sequences, making retrieval sensitive to the choice of serialization even when table semantics remain…
On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems
Md Ibrahim Ibne Alam, Parikshit Ram, Soham Dan +2
Large Language Models (LLMs) have been observed to perform well on a wide range of downstream tasks when fine-tuned on domain-specific data. However, such data may not be readily a…
Few-shot Policy (de)composition in Conversational Question Answering
Kyle Erwin, Guy Axelrod, Maria Chang +8
The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of…