The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC): Selection of a performance metric for classification probabilities balancing diverse science goals

Abstract

Classification of transient and variable light curves is an essential step in using astronomical observations to develop an understanding of their underlying physical processes. However, upcoming deep photometric surveys, including the Large Synoptic Survey Telescope (LSST), will produce a deluge of low signal-to-noise data for which traditional labeling procedures are inappropriate. Probabilistic classification is more appropriate for the data but are incompatible with the traditional metrics used on deterministic classifications. Furthermore, large survey collaborations intend to use these classification probabilities for diverse science objectives, indicating a need for a metric that balances a variety of goals. We describe the process used to develop an optimal performance metric for an open classification challenge that seeks probabilistic classifications and must serve many scientific interests. The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC) is an open competition aiming to identify promising techniques for obtaining classification probabilities of transient and variable objects by engaging a broader community both within and outside astronomy. Using mock classification probability submissions emulating archetypes of those anticipated of PLAsTiCC, we compare the sensitivity of metrics of classification probabilities under various weighting schemes, finding that they yield qualitatively consistent results. We choose as a metric for PLAsTiCC a weighted modification of the cross-entropy because it can be meaningfully interpreted. Finally, we propose extensions of our methodology to ever more complex challenge goals and suggest some guiding principles for approaching the choice of a metric of probabilistic classifications.

References in corpus (20)

• The NumPy array: a structure for efficient numerical computation
• Rotation-invariant convolutional neural networks for galaxy morphology prediction
• Gravity Spy: Integrating Advanced LIGO Detector Characterization, Machine Learning, and Citizen Science
• SiFTO: An Empirical Method for Fitting SNe Ia Light Curves
• Star-galaxy Classification Using Deep Convolutional Neural Networks
• Ensemble modeling of CMEs using the WSA-ENLIL+Cone model
• Glitch Classification and Clustering for LIGO with Deep Transfer Learning
• Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning
• Results from the Supernova Photometric Classification Challenge
• Measuring Dark Energy Properties with Photometrically Classified Pan-STARRS Supernovae. II. Cosmological Parameters
• Deep-HiTS: Rotation Invariant Convolutional Neural Network for Transient Detection
• Radio Galaxy Zoo: ClaRAN - A Deep Learning Classifier for Radio Morphologies
• Machine Learning-based Brokers for Real-time Classification of the LSST Alert Stream
• Deep Learning Classification in Asteroseismology
• Deep Learning Classification in Asteroseismology Using an Improved Neural Network: Results on 15000 Kepler Red Giants and Applications to K2 and TESS Data
• Photometric classification of type Ia supernovae in the SuperNova Legacy Survey with supervised learning
• Detection of Bars in Galaxies using a Deep Convolutional Neural Network
• Detecting Solar-like Oscillations in Red Giants with Deep Learning
• Machine-learning Selection of Optical Transients in Subaru/Hyper Suprime-Cam Survey
• Approximating photo-$z$ PDFs for large surveys

Cited by in corpus (11)

• Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation
• The Young Supernova Experiment: Survey Goals, Overview, and Operations
• Exoplanet Validation with Machine Learning: 50 new validated Kepler planets
• ParSNIP: Generative Models of Transient Light Curves with Physics-Enabled Deep Learning
• Machine Learning for Observational Cosmology
• Photometric classification of HSC transients using machine learning
• On the cosmological performance of photometrically classified supernovae with machine learning
• Impact of Rubin Observatory cadence choices on supernovae photometric classification
• Identification of problematic epochs in astronomical time series through transfer learning
• Considerations for optimizing photometric classification of supernovae from the Rubin Observatory
• Statistical characterization and classification of astronomical transients with Machine Learning in the era of the Vera C. Rubin Observatory