56 citations · 122 across the 22 of their papers we have counts for
25 papers
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3
Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up…
EvoOtter: Evolutionary Reproduction Test Generator
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descri…
Reproduction Test Generation for Java SWE Issues
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Given an issue on a software repository, a reproduction test confirms its presence in the code before it gets fixed and its absence after. Reproduction tests provide crucial execut…
Resolving Java Code Repository Issues with iSWE Agent
Jatin Ganhotra, Sami Serhan, Antonio Abu Nassar +3
Resolving issues on code repositories is an important part of software engineering. Various recent systems automatically resolve issues using large language models and agents, ofte…
Investigating Test Overfitting on SWE-bench
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests b…
Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, s…
OASBuilder: Generating OpenAPI Specifications from Online API Documentation with Large Language Models
Koren Lazar, Matan Vetzler, Kiran Kate +8
AI agents and business automation tools interacting with external web services require standardized, machine-readable information about their APIs in the form of API specifications…
ACE: Towards A High-Assurance Isolated Virtualization Environment for RISC-V
Wojciech Ozga, Guerney D. H. Hunt, Michael V. Le +5
Confidential computing has proven its value in cloud environments, but its potential for securing edge and high-end embedded systems (e.g., automotive controllers, cryptographic ac…
Otter: Generating Tests from Issues to Validate SWE Patches
Toufique Ahmed, Jatin Ganhotra, Rangeet Pan +3
While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch…
TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?
Toufique Ahmed, Martin Hirzel, Rangeet Pan +2
Test-driven development (TDD) is the practice of writing tests first and coding later, and the proponents of TDD expound its numerous benefits. For instance, given an issue on a so…
Towards a Formally Verified Security Monitor for VM-based Confidential Computing
Wojciech Ozga, Guerney D. H. Hunt, Michael V. Le +2
Confidential computing is a key technology for isolating high-assurance applications from the large amounts of untrusted code typical in modern systems. Existing confidential compu…
Transformer Models for Type Inference in the Simply Typed Lambda Calculus: A Case Study in Deep Learning for Code
Brando Miranda, Avi Shinnar, Vasily Pestun +1
Despite a growing body of work at the intersection of deep learning and formal languages, there has been relatively little systematic exploration of transformer models for reasonin…
Navigating Ensemble Configurations for Algorithmic Fairness
Michael Feffer, Martin Hirzel, Samuel C. Hoffman +3
Bias mitigators can improve algorithmic fairness in machine learning models, but their effect on fairness is often not stable across data splits. A popular approach to train more s…
Translating Canonical SQL to Imperative Code in Coq
Véronique Benzaken, Évelyne Contejean, Mohammed Houssem Hachmaoui +4
SQL is by far the most widely used and implemented query language. Yet, on some key features, such as correlated queries and NULL value semantics, many implementations diverge or c…
Formalization of a Stochastic Approximation Theorem
Koundinya Vajjha, Barry Trager, Avraham Shinnar +1
Stochastic approximation algorithms are iterative procedures which are used to approximate a target value in an environment where the target is unknown and direct observations are…
An Empirical Study of Modular Bias Mitigators and Ensembles
Michael Feffer, Martin Hirzel, Samuel C. Hoffman +3
There are several bias mitigators that can reduce algorithmic bias in machine learning models but, unfortunately, the effect of mitigators on fairness is often not stable when meas…
CertRL: Formalizing Convergence Proofs for Value and Policy Iteration in Coq
Koundinya Vajjha, Avraham Shinnar, Vasily Pestun +2
Reinforcement learning algorithms solve sequential decision-making problems in probabilistic environments by optimizing for long-term reward. The desire to use reinforcement learni…
Lale: Consistent Automated Machine Learning
Guillaume Baudart, Martin Hirzel, Kiran Kate +2
Automated machine learning makes it easier for data scientists to develop pipelines by searching over possible choices for hyperparameters, algorithms, and even pipeline topologies…
Type Safety with JSON Subschema
Andrew Habib, Avraham Shinnar, Martin Hirzel +1
JSON is a popular data format used pervasively in web APIs, cloud computing, NoSQL databases, and increasingly also machine learning. JSON Schema is a language for declaring the st…
Type-Driven Automated Learning with Lale
Martin Hirzel, Kiran Kate, Avraham Shinnar +2
Machine-learning automation tools, ranging from humble grid-search to hyperopt, auto-sklearn, and TPOT, help explore large search spaces of possible pipelines. Unfortunately, each…
Yaps: Python Frontend to Stan
Guillaume Baudart, Martin Hirzel, Kiran Kate +2
Stan is a popular probabilistic programming language with a self-contained syntax and semantics that is close to graphical models. Unfortunately, existing embeddings of Stan in Pyt…
Compiling Stan to Generative Probabilistic Languages and Extension to Deep Probabilistic Programming
Guillaume Baudart, Javier Burroni, Martin Hirzel +2
Stan is a probabilistic programming language that is popular in the statistics community, with a high-level syntax for expressing probabilistic models. Stan differs by nature from…
Ariadne: Analysis for Machine Learning Program
Julian Dolby, Avraham Shinnar, Allison Allain +1
Machine learning has transformed domains like vision and translation, and is now increasingly used in science, where the correctness of such code is vital. Python is popular for ma…
Semantics of (Resilient) X10
Silvia Crafa, David Cunningham, Vijay Saraswat +2
We present a formal small-step structural operational semantics for a large fragment of X10, unifying past work. The fragment covers multiple places, mutable objects on the heap, s…