184 citations · 455 across the 23 of their papers we have counts for
23 papers
Large Language Models in Fault Localisation
Yonghao Wu, Zheng Li, Jie M. Zhang +3
Large Language Models (LLMs) have shown promise in multiple software engineering tasks including code generation, program repair, code summarisation, and test generation. Fault loc…
COCO: Testing Code Generation Systems via Concretized Instructions
Ming Yan, Junjie Chen, Jie M. Zhang +3
Code generation systems have been extensively developed in recent years to generate source code based on natural language instructions. However, despite their advancements, these s…
An Empirical Study of the Non-determinism of ChatGPT in Code Generation
Shuyin Ouyang, Jie M. Zhang, Mark Harman +1
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highl…
Fairness Improvement with Multiple Protected Attributes: How Far Are We?
Zhenpeng Chen, Jie M. Zhang, Federica Sarro +1
Existing research mostly improves the fairness of Machine Learning (ML) software regarding a single protected attribute at a time, but this is unrealistic given that many users hav…
Simulation-Driven Automated End-to-End Test and Oracle Inference
Shreshth Tuli, Kinga Bojarczuk, Natalija Gucevska +3
This is the first work to report on inferential testing at scale in industry. Specifically, it reports the experience of automated testing of integrity systems at Meta. We built an…
Keeping Mutation Test Suites Consistent and Relevant with Long-Standing Mutants
Milos Ojdanic, Mike Papadakis, Mark Harman
Mutation testing has been demonstrated to be one of the most powerful fault-revealing tools in the tester's tool kit. Much previous work implicitly assumed it to be sufficient to r…
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Zhenpeng Chen, Jie M. Zhang, Max Hort +2
Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedica…
Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Max Hort, Zhenpeng Chen, Jie M. Zhang +2
This paper provides a comprehensive survey of bias mitigation methods for achieving fairness in Machine Learning (ML) models. We collect a total of 341 publications concerning bias…
A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers
Zhenpeng Chen, Jie M. Zhang, Federica Sarro +1
Software bias is an increasingly important operational concern for software engineers. We present a large-scale, comprehensive empirical study of 17 representative bias mitigation…
Leveraging Automated Unit Tests for Unsupervised Code Translation
Baptiste Roziere, Jie M. Zhang, Francois Charton +3
With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsupervised machine…
An Empirical Study on Failed Error Propagation in Java Programs with Real Faults
Gunel Jahangirova, David Clark, Mark Harman +1
During testing, developers can place oracles externally or internally with respect to a method. Given a faulty execution state, i.e., one that differs from the expected one, an ora…
FrUITeR: A Framework for Evaluating UI Test Reuse
Yixue Zhao, Justin Chen, Adriana Sejfia +5
UI testing is tedious and time-consuming due to the manual effort required. Recent research has explored opportunities for reusing existing UI tests from an app to automatically ge…
Ownership at Large -- Open Problems and Challenges in Ownership Management
John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk +10
Software-intensive organizations rely on large numbers of software assets of different types, e.g., source-code files, tables in the data warehouse, and software configurations. Wh…
WES: Agent-based User Interaction Simulation on Real Infrastructure
John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk +9
We introduce the Web-Enabled Simulation (WES) research agenda, and describe FACEBOOK's WW system. We describe the application of WW to reliability, integrity and privacy at FACEBOO…
FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. A Case Study on Mutation Testing and Program Repair
Maxime Cordy, Renaud Rwemalika, Mike Papadakis +1
Much research on software testing makes an implicit assumption that test failures are deterministic such that they always witness the presence of the same defects. However, this as…
Automatic Testing and Improvement of Machine Translation
Zeyu Sun, Jie M. Zhang, Mark Harman +2
This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic…
A Survey of Constrained Combinatorial Testing
Huayao Wu, Changhai Nie, Justyna Petke +2
Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an…
Machine Learning Testing: Survey, Landscapes and Horizons
Jie M. Zhang, Mark Harman, Lei Ma +1
This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairnes…
Sub-Turing Islands in the Wild
Earl T. Barr, David W. Binkley, Mark Harman +1
Recently, there has been growing debate as to whether or not static analysis can be truly sound. In spite of this concern, research on techniques seeking to at least partially answ…
Model Validation Using Mutated Training Labels: An Exploratory Study
Jie M. Zhang, Mark Harman, Benjamin Guedj +2
We introduce an exploratory study on Mutation Validation (MV), a model validation method using mutated training labels for supervised learning. MV mutates training data labels, ret…
Indexing Operators to Extend the Reach of Symbolic Execution
Earl T. Barr, David Clark, Mark Harman +1
Traditional program analysis analyses a program language, that is, all programs that can be written in the language. There is a difference, however, between all possible programs t…
A Study of Bug Resolution Characteristics in Popular Programming Languages
Jie M. Zhang, Feng Li, Dan Hao +4
This paper presents a large-scale study that investigates the bug resolution characteristics among popular Github projects written in different programming languages. We explore co…
Using Genetic Programming to Model Software
W. B. Langdon, M. Harman
We study a generic program to investigate the scope for automatically customising it for a vital current task, which was not considered when it was first written. In detail, we sho…