Results from the Supernova Photometric Classification Challenge

Abstract

We report results from the Supernova Photometric Classification Challenge (SNPCC), a publicly released mix of simulated supernovae (SNe), with types (Ia, Ibc, and II) selected in proportion to their expected rate. The simulation was realized in the griz filters of the Dark Energy Survey (DES) with realistic observing conditions (sky noise, point-spread function and atmospheric transparency) based on years of recorded conditions at the DES site. Simulations of non-Ia type SNe are based on spectroscopically confirmed light curves that include unpublished non-Ia samples donated from the Carnegie Supernova Project (CSP), the Supernova Legacy Survey (SNLS), and the Sloan Digital Sky Survey-II (SDSS-II). A spectroscopically confirmed subset was provided for training. We challenged scientists to run their classification algorithms and report a type and photo-z for each SN. Participants from 10 groups contributed 13 entries for the sample that included a host-galaxy photo-z for each SN, and 9 entries for the sample that had no redshift information. Several different classification strategies resulted in similar performance, and for all entries the performance was significantly better for the training subset than for the unconfirmed sample. For the spectroscopically unconfirmed subset, the entry with the highest average figure of merit for classifying SNe~Ia has an efficiency of 0.96 and an SN~Ia purity of 0.79. As a public resource for the future development of photometric SN classification and photo-z estimators, we have released updated simulations with improvements based on our experience from the SNPCC, added samples corresponding to the Large Synoptic Survey Telescope (LSST) and the SDSS, and provided the answer keys so that developers can evaluate their own analysis.

accepted by PASP

References in corpus (46)

• Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant
• The Sloan Digital Sky Survey: Technical Summary
• Type Ia Supernova Discoveries at z>1 From the Hubble Space Telescope: Evidence for Past Deceleration and Constraints on Dark Energy Evolution
• The Supernova Legacy Survey: Measurement of Omega_M, Omega_Lambda and w from the First Year Data Set
• Cosmological Results from High-z Supernovae
• Measurements of the Cosmological Parameters Omega and Lambda from the First 7 Supernovae at z >= 0.35
• Observational Constraints on the Nature of the Dark Energy: First Cosmological Results from the ESSENCE Supernova Survey
• LSST Science Book, Version 2.0
• SALT2: using distant supernovae to improve the use of Type Ia supernovae as distance indicators
• Improved Distances to Type Ia Supernovae with Multicolor Light Curve Shapes: MLCS2k2
• The death of massive stars - I. Observational constraints on the progenitors of type II-P supernovae
• First-year Sloan Digital Sky Survey-II (SDSS-II) Supernova Results: Hubble Diagram and Cosmological Parameters
• The Sloan Digital Sky Survey-II Supernova Survey: Technical Summary
• SNANA: A Public Software Package for Supernova Analysis
• SiFTO: An Empirical Method for Fitting SNe Ia Light Curves
• The Sloan Digital Sky Survey-II Supernova Survey: Search Algorithm and Follow-up Observations
• A Comparative Study of the Absolute-Magnitude Distributions of Supernovae
• Optical and Infrared Spectroscopy of SN 1999ee and SN 1999ex
• Controlling the False Discovery Rate in Astrophysical Data Analysis
• A Galaxy Photometric Redshift Catalog for the Sloan Digital Sky Survey Data Release 6
• The Core-collapse rate from the Supernova Legacy Survey
• A Measurement of the Rate of type-Ia Supernovae at Redshift $z\approx$ 0.1 from the First Season of the SDSS-II Supernova Survey
• Type IIP Supernovae as Cosmological Probes: A SEAM Distance to SN 1999em
• The Carnegie Supernova Project: First Near-Infrared Hubble Diagram to z~0.7
• The Type Ia Supernova Rate at z ~0.5 from the Supernova Legacy Survey
• High Redshift Supernovae in the Hubble Deep Field
• Photometric selection of high-redshift type Ia supernovae
• Not Color Blind: Using Multiband Photometry to Classify Supernovae
• Supernovae in the Subaru Deep Field: An Initial Sample, and Type Ia Rate, out to Redshift 1.6
• Type II-P Supernovae from the SDSS-II Supernova Survey and the Standardized Candle Method
• Bayesian Estimation Applied to Multiple Species: Towards cosmology with a million supernovae
• GRB 020410: A Gamma-Ray Burst Afterglow Discovered by its Supernova Light
• A New Determination of the High Redshift Type Ia Supernova Rates with the Hubble Space Telescope Advanced Camera for Surveys
• The Rate of Type Ia Supernovae at High Redshift
• Photometric Redshift Error Estimators
• Bayesian Single-Epoch Photometric Classification of Supernovae
• Photometric Estimates of Redshifts and Distance Moduli for Type Ia Supernovae
• A Probabilistic Approach to Classifying Supernovae Using Photometric Information
• Fuzzy Supernova Templates I: Classification
• Exploiting Low-Dimensional Structure in Astronomical Spectra
• Photometric Identification of Type Ia Supernovae at Moderate Redshift
• Photometric Identification of Young Stripped-Core Supernovae
• Supernova Photometric Classification Challenge
• Selection of high-z supernovae candidates
• Fuzzy Supernova Templates II: Parameter Estimation
• Dark Energy Survey Supernovae: Simulations and Survey Strategy

Cited by in corpus (112)

• Observational Probes of Cosmic Acceleration
• SNLS3: Constraints on Dark Energy Combining the Supernova Legacy Survey Three Year Data with Other Probes
• On Machine-Learned Classification of Variable Stars with Sparse and Noisy Time-Series Data
• Correcting Type Ia Supernova Distances for Selection Biases and Contamination in Photometrically Identified Samples
• The Data Release of the Sloan Digital Sky Survey-II Supernova Survey
• Photometric Supernova Classification With Machine Learning
• SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification
• Simulations of the WFIRST Supernova Survey and Forecasts of Cosmological Constraints
• Supernova Simulations and Strategies For the Dark Energy Survey
• The Difference Imaging Pipeline for the Transient Search in the Dark Energy Survey
• Measuring Dark Energy Properties with Photometrically Classified Pan-STARRS Supernovae. II. Cosmological Parameters
• Photometric SN Ia Candidates from the Three-Year SDSS-II SN Survey Data
• RAPID: Early Classification of Explosive Transients using Deep Learning
• Models and Simulations for the Photometric LSST Astronomical Time Series Classification Challenge (PLAsTiCC)
• A Spectroscopic Study of Type Ibc Supernova Host Galaxies from Untargeted Surveys
• The SDSS-II Supernova Survey: Parameterizing the Type Ia Supernova Rate as a Function of Host Galaxy Properties
• Cosmology with Photometrically-Classified Type Ia Supernovae from the SDSS-II Supernova Survey
• Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation
• Supernova rates from the SUDARE VST-Omegacam search. I
• Deep Recurrent Neural Networks for Supernovae Classification
• The Young Supernova Experiment: Survey Goals, Overview, and Operations
• Machine Learning-based Brokers for Real-time Classification of the LSST Alert Stream
• First Cosmology Results using Type Ia Supernova from the Dark Energy Survey: Simulations to Correct Supernova Distance Biases
• Spectroscopic Properties of Star-Forming Host Galaxies and Type Ia Supernova Hubble Residuals in a Nearly Unbiased Sample
• SuperRAENN: A Semi-supervised Supernova Photometric Classification Pipeline Trained on Pan-STARRS1 Medium Deep Survey Supernovae
• The Discovery of the Most Distant Known Type Ia Supernova at Redshift 1.914
• Enabling real-time multi-messenger astrophysics discoveries with deep learning
• Pair-Instability Supernovae at the Epoch of Reionization
• How Many Kilonovae Can Be Found in Past, Present, and Future Survey Datasets?
• Optimizing spectroscopic follow-up strategies for supernova photometric classification with active learning
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• The Rate of Supernovae at Redshift 0.1-1.0 - the Stockholm VIMOS Supernova Survey IV
• Kernel PCA for type Ia supernovae photometric classification
• Spectrophotometric templates for core collapse supernovae and their application in simulations of time-domain surveys
• Supernova Photometric Classification Pipelines Trained on Spectroscopically Classified Supernovae from the Pan-STARRS1 Medium-Deep Survey
• Active Learning to Overcome Sample Selection Bias: Application to Photometric Variable Star Classification
• Photometric classification of type Ia supernovae in the SuperNova Legacy Survey with supervised learning
• A simple and robust method for automated photometric classification of supernovae using neural networks
• Three Gravitationally Lensed Supernovae Behind CLASH Galaxy Clusters
• Photometric selection of Type Ia supernovae in the Supernova Legacy Survey
• Cosmological Results from the RAISIN Survey: Using Type Ia Supernovae in the Near Infrared as a Novel Path to Measure the Dark Energy Equation of State
• Extending Supernova Spectral Templates for Next-Generation Space Telescope Observations
• Measuring the Properties of Dark Energy with Photometrically Classified Pan-STARRS Supernovae. I. Systematic Uncertainty from Core-Collapse Supernova Contamination
• Turning Gravitationally Lensed Supernovae into Cosmological Probes
• Machine Learning Classification of SDSS Transient Survey Images
• PELICAN: deeP architecturE for the LIght Curve ANalysis
• Semi-supervised Learning for Photometric Supernova Classification
• GHOST: Using Only Host Galaxy Information to Accurately Associate and Distinguish Supernovae
• Probing the stellar initial mass function with high-$z$ supernovae
• A Search for Kilonovae in the Dark Energy Survey
• Constraints on the Physical Properties of GW190814 through Simulations based on DECam Follow-up Observations by the Dark Energy Survey
• Anomaly Detection in the Open Supernova Catalog
• STACCATO: A Novel Solution to Supernova Photometric Classification with Biased Training Sets
• The Dark Energy Survey Supernova Program: Modelling selection efficiency and observed core collapse supernova contamination
• Machine-assisted discovery of relationships in astronomy
• Classifying Supernovae Using Only Galaxy Data
• Projected Cosmological Constraints from Strongly Lensed Supernovae with the Roman Space Telescope
• The Dark Energy Survey 5-year photometrically identified Type Ia Supernovae
• A Type II Supernova Hubble diagram from the CSP-I, SDSS-II, and SNLS surveys
• Machine Learning for Observational Cosmology
• Defining Photometric Peculiar Type Ia Supernovae
• Fink: early supernovae Ia classification using active learning
• The Discovery of a Gravitationally Lensed Supernova Ia at Redshift 2.22
• J-GEM Follow-Up Observations to Search for an Optical Counterpart of The First Gravitational Wave Source GW150914
• The Dark Energy Survey Supernova Program: Cosmological biases from supernova photometric classification
• The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC): Selection of a performance metric for classification probabilities balancing diverse science goals
• zBEAMS: A unified solution for supernova cosmology with redshift uncertainties
• Photometric classification and redshift estimation of LSST Supernovae
• Photometric Classification of Early-Time Supernova Lightcurves with SCONE
• A Search for Optical Emission from Binary-Black-Hole Merger GW170814 with the Dark Energy Camera
• The Type Ia supernovae rate with Subaru/XMM-Newton Deep Survey
• Nonparametric Transient Classification using Adaptive Wavelets
• Density Based Outlier Scoring on Kepler Data
• Star formation and morphological properties of galaxies in the Pan-STARRS $3\pi$ survey- I. A machine learning approach to galaxy and supernova classification
• Linking Extragalactic Transients and their Host Galaxy Properties: Transient Sample, Multi-Wavelength Host Identification, and Database Construction
• Distance Probes of Dark Energy
• Identifying complex sources in large astronomical data using a coarse-grained complexity measure
• A DECam Search for Explosive Optical Transients Associated with IceCube Neutrinos
• Parameter Estimation with BEAMS in the presence of biases and correlations
• SNe Ia as a cosmological probe
• DeepZipper: A Novel Deep Learning Architecture for Lensed Supernovae Identification
• HAFFET: Hybrid Analytic Flux FittEr for Transients
• An Empirical Fitting Method for Type Ia Supernova Light Curves: A Case Study of SN 2011fe
• Assessment of the Systematic Uncertainties in the Cosmological Analysis of the SDSS Supernovae Photometric Sample
• Transient Classifiers for Fink: Benchmarks for LSST
• Identifying anomalous radio sources in the EMU Pilot Survey using a complexity-based approach
• Optimising a magnitude-limited spectroscopic training sample for photometric classification of supernovae
• On the cosmological performance of photometrically classified supernovae with machine learning
• SNAD Transient Miner: Finding Missed Transient Events in ZTF DR4 using k-D trees
• A Spectroscopic Model of the Type Ia Supernova--Host Galaxy Mass Correlation from SALT3
• Understanding of the properties of neural network approaches for transient light curve approximations
• Impact of Rubin Observatory cadence choices on supernovae photometric classification
• SNGuess: A method for the selection of young extragalactic transients
• Understanding Type Ia Supernova Distance Biases by Simulating Spectral Variations
• BEAMS: separating the wheat from the chaff in supernova analysis
• A search for gravitationally lensed supernovae within the Zwicky Transient Facility public survey
• Data challenges as a tool for time-domain astronomy
• Photometric type Ia supernova surveys in narrow band filters
• The discovery and classification of 16 supernovae at high redshifts in ELAIS-S1 - the Stockholm VIMOS Supernova Survey II
• Stratified Learning: A General-Purpose Statistical Method for Improved Learning under Covariate Shift
• Forecasting Supernova Observations with the CSST: I. Photometric Samples
• Type Ia Supernovae Selection and Forecast of Cosmology Constraints for the Dark Energy Survey
• Propagating Uncertainties in the SALT3 Model Training Process to Cosmological Constraints
• Real-Time Active Learning for optimised spectroscopic follow-up: Enhancing early SN Ia classification with the Fink broker
• Using Random Forest Machine Learning Algorithms in Binary Supernovae Classification
• Building spectral templates and reconstructing parameters for core collapse supernovae with CASTOR
• The Case for Space: Estimating Precise Time Delays from Ground- and Space-Based Observations of Lensed Supernovae with Glimpse
• Statistical characterization and classification of astronomical transients with Machine Learning in the era of the Vera C. Rubin Observatory
• Considerations for optimizing photometric classification of supernovae from the Rubin Observatory
• SCONE: Supernova Classification with a Convolutional Neural Network
• A DESGW Search for the Electromagnetic Counterpart to the LIGO/Virgo Gravitational Wave Binary Neutron Star Merger Candidate S190510g
• DeepZipper II: Searching for Lensed Supernovae in Dark Energy Survey Data with Deep Learning