The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC): Selection of a performance metric for classification probabilities balancing diverse science goals
arXiv:1809.11145 · doi:10.3847/1538-3881/ab3a2f
Abstract
Classification of transient and variable light curves is an essential step in using astronomical observations to develop an understanding of their underlying physical processes. However, upcoming deep photometric surveys, including the Large Synoptic Survey Telescope (LSST), will produce a deluge of low signal-to-noise data for which traditional labeling procedures are inappropriate. Probabilistic classification is more appropriate for the data but are incompatible with the traditional metrics used on deterministic classifications. Furthermore, large survey collaborations intend to use these classification probabilities for diverse science objectives, indicating a need for a metric that balances a variety of goals. We describe the process used to develop an optimal performance metric for an open classification challenge that seeks probabilistic classifications and must serve many scientific interests. The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC) is an open competition aiming to identify promising techniques for obtaining classification probabilities of transient and variable objects by engaging a broader community both within and outside astronomy. Using mock classification probability submissions emulating archetypes of those anticipated of PLAsTiCC, we compare the sensitivity of metrics of classification probabilities under various weighting schemes, finding that they yield qualitatively consistent results. We choose as a metric for PLAsTiCC a weighted modification of the cross-entropy because it can be meaningfully interpreted. Finally, we propose extensions of our methodology to ever more complex challenge goals and suggest some guiding principles for approaching the choice of a metric of probabilistic classifications.
References in corpus (38)
- The NumPy array: a structure for efficient numerical computation
- Rotation-invariant convolutional neural networks for galaxy morphology prediction
- Gravity Spy: Integrating Advanced LIGO Detector Characterization, Machine Learning, and Citizen Science
- K-corrections and Extinction Corrections for Type Ia Supernovae
- SiFTO: An Empirical Method for Fitting SNe Ia Light Curves
- Star-galaxy Classification Using Deep Convolutional Neural Networks
- Photometric Supernova Classification With Machine Learning
- Ensemble modeling of CMEs using the WSA-ENLIL+Cone model
- Deep Transfer Learning: A new deep learning glitch classification method for advanced LIGO
- Glitch Classification and Clustering for LIGO with Deep Transfer Learning
- Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning
- Results from the Supernova Photometric Classification Challenge
- The Third Gravitational Lensing Accuracy Testing (GREAT3) Challenge Handbook
- Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era
- Measuring Dark Energy Properties with Photometrically Classified Pan-STARRS Supernovae. II. Cosmological Parameters
- Photometric SN Ia Candidates from the Three-Year SDSS-II SN Survey Data
- Deep-HiTS: Rotation Invariant Convolutional Neural Network for Transient Detection
- K2 Variable Catalogue II: Machine Learning Classification of Variable Stars and Eclipsing Binaries in K2 Fields 0-4
- Radio Galaxy Zoo: ClaRAN - A Deep Learning Classifier for Radio Morphologies
- UNITY: Confronting Supernova Cosmology's Statistical and Systematic Uncertainties in a Unified Bayesian Framework
- Machine Learning-based Brokers for Real-time Classification of the LSST Alert Stream
- Construction of a Calibrated Probabilistic Classification Catalog: Application to 50k Variable Sources in the All-Sky Automated Survey
- Deep Learning Classification in Asteroseismology
- Deep Learning Classification in Asteroseismology Using an Improved Neural Network: Results on 15000 Kepler Red Giants and Applications to K2 and TESS Data
- Kernel PCA for type Ia supernovae photometric classification
- Separation of pulsar signals from noise with supervised machine learning algorithms
- Bayesian High-Redshift Quasar Classification from Optical and Mid-IR Photometry
- Photometric classification of type Ia supernovae in the SuperNova Legacy Survey with supervised learning
- Detection of Bars in Galaxies using a Deep Convolutional Neural Network
- Gravitational Lensing Accuracy Testing 2010 (GREAT10) Challenge Handbook
- A Hybrid Ensemble Learning Approach to Star-Galaxy Classification
- Automated Probabilistic Classification of Transients and Variables
- Detecting Solar-like Oscillations in Red Giants with Deep Learning
- Machine-learning Selection of Optical Transients in Subaru/Hyper Suprime-Cam Survey
- Approximating photo- PDFs for large surveys
- Statistical Classification Techniques for Photometric Supernova Typing
- zBEAMS: A unified solution for supernova cosmology with redshift uncertainties
- Detecting Quasars in Large-Scale Astronomical Surveys
Cited by in corpus (11)
- Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation
- The Young Supernova Experiment: Survey Goals, Overview, and Operations
- Exoplanet Validation with Machine Learning: 50 new validated Kepler planets
- ParSNIP: Generative Models of Transient Light Curves with Physics-Enabled Deep Learning
- Machine Learning for Observational Cosmology
- Photometric classification of HSC transients using machine learning
- On the cosmological performance of photometrically classified supernovae with machine learning
- Impact of Rubin Observatory cadence choices on supernovae photometric classification
- Identification of problematic epochs in astronomical time series through transfer learning
- Considerations for optimizing photometric classification of supernovae from the Rubin Observatory
- Statistical characterization and classification of astronomical transients with Machine Learning in the era of the Vera C. Rubin Observatory