Publications (45)
Enhancing Testing at Meta with Rich-State Simulated Populations
Nadia Alshahwan, Arianna Blasi, Kinga Bojarczuk +11
This paper reports the results of the deployment of Rich-State Simulated Populations at Meta for both automated and manual testing. We use simulated users (aka test users) to mimic…
Large Language Models in Fault Localisation
Yonghao Wu, Zheng Li, Jie M. Zhang +3
Large Language Models (LLMs) have shown promise in multiple software engineering tasks including code generation, program repair, code summarisation, and test generation. Fault loc…
HotBugs.jar: A Benchmark of Hot Fixes for Time-Critical Bugs
Carol Hanna, Federica Sarro, Mark Harman +1
Hot fixes are urgent, unplanned changes deployed to production systems to address time-critical issues. Despite their importance, no existing evaluation benchmark focuses specifica…
Fairness Improvement with Multiple Protected Attributes: How Far Are We?
Zhenpeng Chen, Jie M. Zhang, Federica Sarro +1
Existing research mostly improves the fairness of Machine Learning (ML) software regarding a single protected attribute at a time, but this is unrealistic given that many users hav…
An Empirical Study of the Non-determinism of ChatGPT in Code Generation
Shuyin Ouyang, Jie M. Zhang, Mark Harman +1
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highl…
FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. A Case Study on Mutation Testing and Program Repair
Maxime Cordy, Renaud Rwemalika, Mike Papadakis +1
Much research on software testing makes an implicit assumption that test failures are deterministic such that they always witness the presence of the same defects. However, this as…
COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation
Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova +3
Code generation systems make each LLM call with a model, a prompt, and decoding settings. However, existing optimization methods usually tune only part of these choices or use one…
COCO: Testing Code Generation Systems via Concretized Instructions
Ming Yan, Junjie Chen, Jie M. Zhang +3
Code generation systems have been extensively developed in recent years to generate source code based on natural language instructions. However, despite their advancements, these s…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
Machine Learning Testing: Survey, Landscapes and Horizons
Jie M. Zhang, Mark Harman, Lei Ma +1
This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairnes…
Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries
Lukas Twist, Jie M. Zhang, Mark Harman +1
Large language models (LLMs) now play a central role in code generation, yet they continue to hallucinate, frequently inventing non-existent libraries. Such library hallucinations…
Assured LLM-Based Software Engineering
Nadia Alshahwan, Mark Harman, Inna Harper +3
In this paper we address the following question: How can we use Large Language Models (LLMs) to improve code independently of a human, while ensuring that the improved code - does…
Leveraging Automated Unit Tests for Unsupervised Code Translation
Baptiste Roziere, Jie M. Zhang, Francois Charton +3
With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsupervised machine…
An Empirical Study on Failed Error Propagation in Java Programs with Real Faults
Gunel Jahangirova, David Clark, Mark Harman +1
During testing, developers can place oracles externally or internally with respect to a method. Given a faulty execution state, i.e., one that differs from the expected one, an ora…
Reducing Hallucination in Enterprise AI Workflows via Hybrid Utility Minimum Bayes Risk (HUMBR)
Chenhao Fang, Jordi Mola, Mark Harman +10
Although LLMs drive automation, it is critical to ensure immense consideration for high-stakes enterprise workflows such as those involving legal matters, risk management, and priv…
Observation-based unit test generation at Meta
Nadia Alshahwan, Mark Harman, Alexandru Marginean +2
TestGen automatically generates unit tests, carved from serialized observations of complex objects, observed during app execution. We describe the development and deployment of Tes…
LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs
Kaibo Liu, Zhenpeng Chen, Yiyang Liu +7
Detecting tricky bugs in plausible programs, those that pass existing test suites yet still contain bugs, remains a significant challenge in software testing. To address this probl…
Automated Unit Test Improvement using Large Language Models at Meta
Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova +6
This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successful…
Prompt Coverage Adequacy
Florian Tambon, Michael Konstantinou, Cedric Richter +3
In recent years, it has become increasingly evident that large language models (LLMs) and autonomous agents raise the level of abstraction in software development by shifting the f…
Model Validation Using Mutated Training Labels: An Exploratory Study
Jie M. Zhang, Mark Harman, Benjamin Guedj +2
We introduce an exploratory study on Mutation Validation (MV), a model validation method using mutated training labels for supervised learning. MV mutates training data labels, ret…
A Comprehensive Study on Large Language Models for Mutation Testing
Bo Wang, Mingda Chen, Ming Deng +4
Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study…
Boosting LLMs for Mutation Generation
Bo Wang, Ming Deng, Mingda Chen +5
LLM-based mutation testing is a promising testing technology, but existing approaches typically rely on a fixed set of mutations as few-shot examples or none at all. This can resul…
A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers
Zhenpeng Chen, Jie M. Zhang, Federica Sarro +1
Software bias is an increasingly important operational concern for software engineers. We present a large-scale, comprehensive empirical study of 17 representative bias mitigation…
WES: Agent-based User Interaction Simulation on Real Infrastructure
John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk +9
We introduce the Web-Enabled Simulation (WES) research agenda, and describe FACEBOOK's WW system. We describe the application of WW to reliability, integrity and privacy at FACEBOO…
Indexing Operators to Extend the Reach of Symbolic Execution
Earl T. Barr, David Clark, Mark Harman +1
Traditional program analysis analyses a program language, that is, all programs that can be written in the language. There is a difference, however, between all possible programs t…
Keeping Mutation Test Suites Consistent and Relevant with Long-Standing Mutants
Milos Ojdanic, Mike Papadakis, Mark Harman
Mutation testing has been demonstrated to be one of the most powerful fault-revealing tools in the tester's tool kit. Much previous work implicitly assumed it to be sufficient to r…
A Study of Bug Resolution Characteristics in Popular Programming Languages
Jie M. Zhang, Feng Li, Dan Hao +4
This paper presents a large-scale study that investigates the bug resolution characteristics among popular Github projects written in different programming languages. We explore co…
Ownership at Large -- Open Problems and Challenges in Ownership Management
John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk +10
Software-intensive organizations rely on large numbers of software assets of different types, e.g., source-code files, tables in the data warehouse, and software configurations. Wh…
Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Max Hort, Zhenpeng Chen, Jie M. Zhang +2
This paper provides a comprehensive survey of bias mitigation methods for achieving fairness in Machine Learning (ML) models. We collect a total of 341 publications concerning bias…
EET: Experience-Driven Early Termination for Cost-Efficient Software Engineering Agents
Yaoqi Guo, Ying Xiao, Jie M. Zhang +4
Software engineering (SE) agents powered by large language models are increasingly adopted in practice, yet they often incur substantial monetary cost. We introduce EET, an experie…
A Study of LLMs' Preferences for Libraries and Programming Languages
Lukas Twist, Mark Harman, Don Syme +4
Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make…
FrUITeR: A Framework for Evaluating UI Test Reuse
Yixue Zhao, Justin Chen, Adriana Sejfia +5
UI testing is tedious and time-consuming due to the manual effort required. Recent research has explored opportunities for reusing existing UI tests from an app to automatically ge…
A Survey of Constrained Combinatorial Testing
Huayao Wu, Changhai Nie, Justyna Petke +2
Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an…
Sub-Turing Islands in the Wild
Earl T. Barr, David W. Binkley, Mark Harman +1
Recently, there has been growing debate as to whether or not static analysis can be truly sound. In spite of this concern, research on techniques seeking to at least partially answ…
Generative AI for Testing of Autonomous Driving Systems: A Survey
Qunying Song, He Ye, Mark Harman +1
Autonomous driving systems (ADS) have been an active area of research, with the potential to deliver significant benefits to society. However, before large-scale deployment on publ…
YATE: The Role of Test Repair in LLM-Based Unit Test Generation
Michael Konstantinou, Renzo Degiovanni, Jie M. Zhang +2
Recent advances in automated test generation utilises language models to produce unit tests. While effective, language models tend to generate many incorrect tests with respect to…
Measuring the Influence of Incorrect Code on Test Generation
Dong Huang, Jie M. Zhang, Mark Harman +2
It is natural to suppose that a Large Language Model is more likely to generate correct test cases when prompted with correct code under test, compared to incorrect code under test…
Automatic Testing and Improvement of Machine Translation
Zeyu Sun, Jie M. Zhang, Mark Harman +2
This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic…
Benchmarking LLMs for Unit Test Generation from Real-World Functions
Dong Huang, Jie M. Zhang, Mark Harman +3
Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively…
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Zhenpeng Chen, Jie M. Zhang, Max Hort +2
Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedica…
Large Language Models for Software Engineering: Survey and Open Problems
Angela Fan, Beliz Gokkaya, Mark Harman +4
This paper provides a survey of the emerging area of Large Language Models (LLMs) for Software Engineering (SE). It also sets out open research challenges for the application of LL…
Simulation-Driven Automated End-to-End Test and Oracle Inference
Shreshth Tuli, Kinga Bojarczuk, Natalija Gucevska +3
This is the first work to report on inferential testing at scale in industry. Specifically, it reports the experience of automated testing of integrity systems at Meta. We built an…
Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
Mark Harman, Peter O'Hearn, Shubho Sengupta
Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world…
Mutation-Guided LLM-based Test Generation at Meta
Christopher Foster, Abhishek Gulati, Mark Harman +5
This paper describes Meta's ACH system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation…
Just-in-Time Catching Test Generation at Meta
Matthew Becker, Yifei Chen, Nicholas Cochran +12
We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hard…