Change Point Detection in Software Performance Testing

Abstract

We describe our process for automatic detection of performance changes for a software product in the presence of noise. A large collection of tests run periodically as changes to our software product are committed to our source repository, and we would like to identify the commits responsible for performance regressions. Previously, we relied on manual inspection of time series graphs to identify significant changes. That was later replaced with a threshold-based detection system, but neither system was sufficient for finding changes in performance in a timely manner. This work describes our recent implementation of a change point detection system built upon the E-Divisive means algorithm. The algorithm produces a list of change points representing significant changes from a given history of performance results. A human reviews the list of change points for actionable changes, which are then triaged for further inspection. Using change point detection has had a dramatic impact on our ability to detect performance changes. Quantitatively, it has dramatically dropped our false positive rate for performance changes, while qualitatively it has made the entire performance evaluation process easier, more productive (ex. catching smaller regressions), and more timely.

Cited by in corpus (8)

• Creating a Virtuous Cycle in Performance Testing at MongoDB
• Automated System Performance Testing at MongoDB
• Evaluating Search-Based Software Microbenchmark Prioritization
• Using Microbenchmark Suites to Detect Application Performance Changes
• The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application Benchmarks
• Automated Identification of Performance Changes at Code Level
• Efficiently Detecting Performance Changes in FaaS Application Releases
• $\mu$OpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics