Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration

Abstract

Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if traceability links between code and tests are not available. This paper introduces Retecs, a new method for automatically learning test case selection and prioritization in CI with the goal to minimize the round-trip time between code commits and developer feedback on failed test cases. The Retecs method uses reinforcement learning to select and prioritize test cases according to their duration, previous last execution and failure history. In a constantly changing environment, where new test cases are created and obsolete test cases are deleted, the Retecs method learns to prioritize error-prone test cases higher under guidance of a reward function and by observing previous CI cycles. By applying Retecs on data extracted from three industrial case studies, we show for the first time that reinforcement learning enables fruitful automatic adaptive test case selection and prioritization in CI and regression testing.

Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration. In Proceedings of 26th International Symposium on Software Testing and Analysis (ISSTA'17) (pp. 12--22). ACM

References in corpus (2)

• Scikit-learn: Machine Learning in Python
• Apprenticeship Learning using Inverse Reinforcement Learning and Gradient Methods

Cited by in corpus (19)

• Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review
• Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts
• Test Prioritization in Continuous Integration Environments
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• Test case prioritization using test case diversification and fault-proneness estimations
• An Approach and Benchmark to Detect Behavioral Changes of Commits in Continuous Integration
• Quantum Approximate Optimization Algorithm for Test Case Optimization
• A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning
• Test Case Minimization with Quantum Annealers
• Neural Network Embeddings for Test Case Prioritization
• GUI Test Migration via Abstraction and Concretization
• Automatic Web Testing using Curiosity-Driven Reinforcement Learning
• SAGA: Summarization-Guided Assert Statement Generation
• Automated System-Level Software Testing of Industrial Networked Embedded Systems
• Dynamic Test Case Prioritization in Industrial Test Result Datasets
• Enhancing Dynamic Symbolic Execution by Automatically Learning Search Heuristics
• Constraint-Guided Test Execution Scheduling: An Experience Report at ABB Robotics
• Multi-Cycle Assignment Problems with Rotational Diversity
• Reinforcement Learning for Test Case Prioritization