Classification uncertainty for transient gravitational-wave noise artefacts with optimised conformal prediction

Abstract

With the increasing use of Machine Learning (ML) algorithms in scientific research comes the need for reliable uncertainty quantification. When taking a measurement it is not enough to provide the result, we also have to declare how confident we are in the measurement. This is also true when the results are obtained from a ML algorithm, and arguably more so since the internal workings of ML algorithms are often less transparent compared to traditional statistical methods. Additionally, many ML algorithms do not provide uncertainty estimates and auxiliary algorithms must be applied. Conformal Prediction (CP) is a framework to provide such uncertainty quantifications for ML point predictors. In this paper, we explore the use and properties of CP applied in the context of glitch classification in gravitational wave astronomy. Specifically, we demonstrate the application of CP to the Gravity Spy glitch classification algorithm. CP makes use of a score function, a nonconformity measure, to convert an algorithm's heuristic notion of uncertainty to a rigorous uncertainty. We use the application on Gravity Spy to explore the performance of different nonconformity measures and optimise them for our application. Our results show that the optimal nonconformity measure depends on the specific application, as well as the metric used to quantify the performance.

References in corpus (23)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Observation of Gravitational Waves from a Binary Black Hole Merger
• Advanced Virgo: a 2nd generation interferometric gravitational wave detector
• Advanced LIGO
• GWTC-3: Compact Binary Coalescences Observed by LIGO and Virgo During the Second Part of the Third Observing Run
• Interferometer design of the KAGRA gravitational wave detector
• Kilonovae
• A guide to LIGO-Virgo detector noise and extraction of transient gravitational-wave signals
• Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (1998)
• Gravity Spy: Integrating Advanced LIGO Detector Characterization, Machine Learning, and Citizen Science
• Open data from the third observing run of LIGO, Virgo, KAGRA and GEO
• Multiresolution techniques for the detection of gravitational-wave bursts
• LIGO Detector Characterization in the Second and Third Observing Runs
• Mitigation of the instrumental noise transient in gravitational-wave data surrounding GW170817
• Subtracting glitches from gravitational-wave detector data during the third observing run
• Environmental Noise in Advanced LIGO Detectors
• Efficient Gravitational-wave Glitch Identification from Environmental Data Through Machine Learning
• Data quality up to the third observing run of Advanced LIGO: Gravity Spy glitch classifications
• Machine-learning classification of astronomical sources: estimating F1-score in the absence of ground truth
• Gravity Spy: Lessons Learned and a Path Forward
• Convolutional Neural Networks for the classification of glitches in gravitational-wave data streams
• GWitchHunters: Machine Learning and citizen science to improve the performance of Gravitational Wave detector
• Class-Conditional Conformal Prediction with Many Classes

Cited by in corpus (2)

• Eccentric or circular? A reanalysis of binary black hole gravitational wave events for orbital eccentricity signatures
• Enhancing gravitational-wave detection: a machine learning pipeline combination approach with robust uncertainty quantification