Robust benchmarking in noisy environments

Abstract

We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter and other environmental fluctuations, and is insensitive to the highly nonideal statistics produced by timing measurements. We construct a model that explains how these strongly nonideal statistics can arise from environmental fluctuations, and also justifies our proposed strategy. We implement this strategy in the BenchmarkTools Julia package, where it is used in production continuous integration (CI) pipelines for developing the Julia language and its ecosystem.

7 pages, 5 figures, Proceedings of the 20th Annual IEEE High Performance Extreme Computing Conference, 2016

Cited by in corpus (10)

• Estimating fractal dimensions: a comparative review and open source implementations
• Structure-preserving approximations of the Serre-Green-Naghdi equations in standard and hyperbolic form
• Copulas.jl: A fully Distributions.jl-compliant copula package
• A Test for FLOPs as a Discriminant for Linear Algebra Algorithms
• Reciprocal Multi-Robot Collision Avoidance with Asymmetric State Uncertainty
• LongMemory.jl: Generating, Estimating, and Forecasting Long Memory Models in Julia
• A Projection Approach to Local Regression with Variable-Dimension Covariates
• Fast Rank-1 NMF for Missing Data with KL Divergence
• A fast, large-scale optimal transport algorithm for holographic beam shaping
• Discriminating Equivalent Algorithms via Relative Performance