A Comparison of Flare Forecasting Methods. II. Benchmarks, Metrics and Performance Results for Operational Solar Flare Forecasting Systems

Abstract

Solar flares are extremely energetic phenomena in our Solar System. Their impulsive, often drastic radiative increases, in particular at short wavelengths, bring immediate impacts that motivate solar physics and space weather research to understand solar flares to the point of being able to forecast them. As data and algorithms improve dramatically, questions must be asked concerning how well the forecasting performs; crucially, we must ask how to rigorously measure performance in order to critically gauge any improvements. Building upon earlier-developed methodology (Barnes et al, 2016, Paper I), international representatives of regional warning centers and research facilities assembled in 2017 at the Institute for Space-Earth Environmental Research, Nagoya University, Japan to - for the first time - directly compare the performance of operational solar flare forecasting methods. Multiple quantitative evaluation metrics are employed, with focus and discussion on evaluation methodologies given the restrictions of operational forecasting. Numerous methods performed consistently above the "no skill" level, although which method scored top marks is decisively a function of flare event definition and the metric used; there was no single winner. Following in this paper series we ask why the performances differ by examining implementation details (Leka et al. 2019, Paper III), and then we present a novel analysis method to evaluate temporal patterns of forecasting errors in (Park et al. 2019, Paper IV). With these works, this team presents a well-defined and robust methodology for evaluating solar flare forecasting methods in both research and operational frameworks, and today's performance benchmarks against which improvements and new methods may be compared.

26 pages, 5 figures, accepted for publication in the Astrophysical Journal Supplement Series

References in corpus (18)

• The Helioseismic and Magnetic Imager (HMI) Vector Magnetic Field Pipeline: Overview and Performance
• Solar Flare Prediction Using SDO/HMI Vector Magnetic Field Data with a Machine-Learning Algorithm
• Toward Reliable Benchmarking of Solar Flare Forecasting Methods
• 25 Years of Self-Organized Criticality: Solar and Astrophysics
• A Comparison of Flare Forecasting Methods, I: Results from the "All-Clear" Workshop
• Solar Flare Prediction Model with Three Machine-Learning Algorithms Using Ultraviolet Brightening and Vector Magnetogram
• The Origin of the Solar Flare Waiting-Time Distribution
• Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning
• The Helioseismic and Magnetic Imager (HMI) Vector Magnetic Field Pipeline: Optimization of the Spectral Line Inversion Code
• A Statistical Solar Flare Forecast Method
• Flare forecasting at the Met Office Space Weather Operations Centre
• Productivity of Solar Flares and Magnetic Helicity Injection in Active Regions
• A Comparison of Flare Forecasting Methods. III. Systematic Behaviors of Operational Solar Flare Forecasting Systems
• Evaluating (and Improving) Estimates of the Solar Radial Magnetic Field Component from Line-of-Sight Magnetograms
• Verification of operational solar flare forecast: Case of Regional Warning Center Japan
• Verification of Space Weather Forecasts issued by the Met Office Space Weather Operations Centre
• Connecting Coronal Mass Ejections to their Solar Active Region Sources: Combining Results from the HELCATS and FLARECAST Projects
• Deterministically Driven Avalanche Models of Solar Flares

Cited by in corpus (35)

• Predicting Solar Flares Using CNN and LSTM on Two Solar Cycles of Active Region Data
• How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events
• Feature ranking of active region source properties in solar flare forecasting and the uncompromised stochasticity of flare occurrence
• A Comparison of Flare Forecasting Methods. III. Systematic Behaviors of Operational Solar Flare Forecasting Systems
• Solar Flare Intensity Prediction with Machine Learning Models
• The Sun and Space Weather
• A Comparison of Flare Forecasting Methods. IV. Evaluating Consecutive-Day Forecasting Patterns
• Operational solar flare prediction model using Deep Flare Net
• Solar Flare Prediction Using Magnetic Field Diagnostics Above the Photosphere
• Implementation paradigm for supervised flare forecasting studies: a deep learning application with video data
• Reliable Probability Forecast of Solar Flares: Deep Flare Net-Reliable (DeFN-R)
• Toward a Next Generation Particle Precipitation Model: Mesoscale Prediction Through Machine Learning (a Case Study and Framework for Progress)
• Flare Forecasting Algorithms Based on High-Gradient Polarity Inversion Lines in Active Regions
• Measuring the magnetic origins of solar flares, CMEs and Space Weather
• Decreasing False Alarm Rates in ML-based Solar Flare Prediction using SDO/HMI Data
• Properties of Flare-Imminent versus Flare-Quiet Active Regions from the Chromosphere through the Corona II: NonParametric Discriminant Analysis Results from the NWRA Classification Infrastructure (NCI)
• Leveraging the Mathematics of Shape for Solar Magnetic Eruption Prediction
• Validation of a wave heated 3D MHD coronal-wind model using Polarized Brightness and EUV observations
• Solar Flares Forecasting Using Time Series and Extreme Gradient Boosting Ensembles
• The characteristics of flare- and CME-productive solar active regions
• Ensemble Forecasting of Major Solar Flares: Methods for Combining Models
• Differences in periodic magnetic helicity injection behaviour between flaring and non-flaring Active Regions: Case Study
• Solar Active Region Magnetogram Image Dataset for Studies of Space Weather
• Comparison of magnetic energy and helicity in coronal jet simulations
• On the differences in the periodic behaviour of magnetic helicity flux in flaring active regions with and without X-class events
• On the possibility to probe the flare productivity of an active region on the early stage of emergence
• Magnetic Helicity Flux across Solar Active Region Photospheres: II. Association of Hemispheric Sign Preference with Flaring Activity during Solar Cycle 24
• Long-Term Evolution of Magnetic Fields in Flaring Active Region NOAA 12673
• Comparative case study of two methods to assess the eruptive potential of selected active regions
• Evaluating Pointing Strategies for Future Solar Flare Missions
• Earth-affecting Solar Transients: A Review of Progresses in Solar Cycle 24
• The Flare Likelihood and Region Eruption Forecasting (FLARECAST) Project: Flare forecasting in the big data & machine learning era
• Testing and Validating Two Morphological Flare Predictors by Logistic Regression Machine Learning
• Forecasting Solar Flares by Data Assimilation in Sandpile Models
• Power-law Indices of EUV Intensity Power Spectrum in Flaring Coronal Active Regions