A Comparison of Flare Forecasting Methods, I: Results from the "All-Clear" Workshop

Abstract

Solar flares produce radiation which can have an almost immediate effect on the near-Earth environment, making it crucial to forecast flares in order to mitigate their negative effects. The number of published approaches to flare forecasting using photospheric magnetic field observations has proliferated, with varying claims about how well each works. Because of the different analysis techniques and data sets used, it is essentially impossible to compare the results from the literature. This problem is exacerbated by the low event rates of large solar flares. The challenges of forecasting rare events have long been recognized in the meteorology community, but have yet to be fully acknowledged by the space weather community. During the interagency workshop on "all clear" forecasts held in Boulder, CO in 2009, the performance of a number of existing algorithms was compared on common data sets, specifically line-of-sight magnetic field and continuum intensity images from MDI, with consistent definitions of what constitutes an event. We demonstrate the importance of making such systematic comparisons, and of using standard verification statistics to determine what constitutes a good prediction scheme. When a comparison was made in this fashion, no one method clearly outperformed all others, which may in part be due to the strong correlations among the parameters used by different methods to characterize an active region. For M-class flares and above, the set of methods tends towards a weakly positive skill score (as measured with several distinct metrics), with no participating method proving substantially better than climatological forecasts.

arXiv commentAccepted for publication in ApJ

References in corpus (13)

• Global Energetics of Thirty-Eight Large Solar Eruptive Events
• Toward Reliable Benchmarking of Solar Flare Forecasting Methods
• A Bayesian Approach to Solar Flare Prediction
• Solar Magnetic Feature Detection and Tracking for Space Weather Monitoring
• A Statistical Solar Flare Forecast Method
• Magnetic Energy and Helicity Budgets in the Active-Region Solar Corona. II. Nonlinear Force-Free Approximation
• Turbulence, Complexity, and Solar Flares
• Are Solar Active Regions with Major Flares More Fractal, Multifractal, or Turbulent than Others?
• Prediction of Solar Flare Size and Time-to-Flare Using Support Vector Machine Regression
• An automated classification approach to ranking photospheric proxies of magnetic energy build-up
• 25 Years of Self-Organized Criticality: Numerical Detection Methods
• Magnetic Nonpotentiality in Photospheric Active Regions as a Predictor of Solar Flares
• Frozen-in Fractals All Around: Inferring the Large Scale Effects of Small-Scale Magnetic Structure

Cited by in corpus (76)

• The origin, early evolution and predictability of solar eruptions
• Solar Flare Prediction Model with Three Machine-Learning Algorithms Using Ultraviolet Brightening and Vector Magnetogram
• Deep Flare Net (DeFN) model for solar flare prediction
• Predicting Solar Flares Using a Long Short-Term Memory Network
• Predicting Solar Flares Using SDO/HMI Vector Magnetic Data Product and Random Forest Algorithm
• A Comparison of Flare Forecasting Methods. II. Benchmarks, Metrics and Performance Results for Operational Solar Flare Forecasting Systems
• Identifying Solar Flare Precursors Using Time Series of SDO/HMI Images and SHARP Parameters
• Relative magnetic helicity as a diagnostic of solar eruptivity
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited
• Predicting Solar Flares Using CNN and LSTM on Two Solar Cycles of Active Region Data
• How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events
• Flare forecasting at the Met Office Space Weather Operations Centre
• The importance of ensemble techniques for operational space weather forecasting
• Feature ranking of active region source properties in solar flare forecasting and the uncompromised stochasticity of flare occurrence
• Successive flux rope eruptions from $\delta$-Sunspots region of NOAA 12673 and associated X-class eruptive flares on 2017 September 6
• A Comparison of Flare Forecasting Methods. III. Systematic Behaviors of Operational Solar Flare Forecasting Systems
• Solar Flare Intensity Prediction with Machine Learning Models
• Prediction of Solar Flares Using Unique Signatures of Magnetic Field Images
• Magnetic helicity and energy budget around large confined and eruptive solar flares
• A Comparison of Flare Forecasting Methods. IV. Evaluating Consecutive-Day Forecasting Patterns
• Operational solar flare prediction model using Deep Flare Net
• A blueprint of state-of-the-art techniques for detecting quasi-periodic pulsations in solar and stellar flares
• Relationships Between Characteristics of the Line-of-Sight Magnetic Field and Solar Flare Forecasts
• On the generation of probabilistic forecasts from deterministic models
• Multi-channel coronal hole detection with convolutional neural networks
• Solar Flare Prediction Using Magnetic Field Diagnostics Above the Photosphere
• Implementation paradigm for supervised flare forecasting studies: a deep learning application with video data
• Connecting Coronal Mass Ejections to their Solar Active Region Sources: Combining Results from the HELCATS and FLARECAST Projects
• Fine-grained Solar Flare Forecasting Based on the Hybrid Convolutional Neural Networks
• Reliable Probability Forecast of Solar Flares: Deep Flare Net-Reliable (DeFN-R)
• Solar Flares Complex Networks
• Toward a Next Generation Particle Precipitation Model: Mesoscale Prediction Through Machine Learning (a Case Study and Framework for Progress)
• Supervised convolutional neural networks for classification of flaring and nonflaring active regions using line-of-sight magnetograms
• Decreasing False Alarm Rates in ML-based Solar Flare Prediction using SDO/HMI Data
• Measuring the magnetic origins of solar flares, CMEs and Space Weather
• Flare Forecasting Algorithms Based on High-Gradient Polarity Inversion Lines in Active Regions
• All-Clear Flare Prediction Using Interval-based Time Series Classifiers
• Signature of the turbulent component of solar dynamo on active region scales and its association with flaring activity
• DeepSun: Machine-Learning-as-a-Service for Solar Flare Prediction
• Self-consistent propagation of flux ropes in realistic coronal simulations
• Leveraging the Mathematics of Shape for Solar Magnetic Eruption Prediction
• Solar Flares Forecasting Using Time Series and Extreme Gradient Boosting Ensembles
• Properties of Flare-Imminent versus Flare-Quiet Active Regions from the Chromosphere through the Corona II: NonParametric Discriminant Analysis Results from the NWRA Classification Infrastructure (NCI)
• Machine Learning Approaches to Solar-Flare Forecasting: Is Complex Better?
• The characteristics of flare- and CME-productive solar active regions
• Ensemble Forecasting of Major Solar Flares: Methods for Combining Models
• Photospheric Shear Flows in Solar Active Regions and Their Relation to Flare Occurrence
• Complex Network for Solar Active Regions
• Differences in periodic magnetic helicity injection behaviour between flaring and non-flaring Active Regions: Case Study
• Current-driven flare and CME models
• Automatic detection of small-scale EUV brightenings observed by the Solar Orbiter/EUI
• Machine learning reveals systematic accumulation of electric current in lead-up to solar flares
• Machine learning as a flaring storm warning machine: Was a warning machine for the September 2017 solar flaring storm possible?
• Global Energetics of Solar Flares. XI. Flare Magnitude Predictions of the GOES-Class
• Global Energetics of Solar Flares. IX. Refined Magnetic Modeling
• Image Quality Assessment for Full-Disk Solar Observations with Generative Adversarial Networks
• On the possibility to probe the flare productivity of an active region on the early stage of emergence
• On the differences in the periodic behaviour of magnetic helicity flux in flaring active regions with and without X-class events
• Flare Productivity of Major Flaring Solar Active Regions: A Time-series Study of Photospheric Magnetic Properties
• Magnetic Helicity Flux across Solar Active Region Photospheres: II. Association of Hemispheric Sign Preference with Flaring Activity during Solar Cycle 24
• A New Space Weather Tool for Identifying Eruptive Active Regions
• Segmentation of spectroscopic images of the low solar atmosphere by the Self Organizing Map technique
• On the evolution of a sub-C class flare: a showcase for the capabilities of the revamped Catania Solar Telescope
• A New Approach to Solar Flare Prediction
• Planetary statistics and forecasting for solar flares
• Solar activity classification based on Mg II spectra: towards classification on compressed data
• Solar Flare Prediction Using Long Short-term Memory (LSTM) and Decomposition-LSTM with Sliding Window Pattern Recognition
• Evaluating Pointing Strategies for Future Solar Flare Missions
• Comparative case study of two methods to assess the eruptive potential of selected active regions
• A new view of the solar interface region from the Interface Region Imaging Spectrograph (IRIS)
• The Flare Likelihood and Region Eruption Forecasting (FLARECAST) Project: Flare forecasting in the big data & machine learning era
• Forecasting Solar Flares by Data Assimilation in Sandpile Models
• The NWRA Classification Infrastructure: Description and Extension to the Discriminant Analysis Flare Forecasting System (DAFFS)
• Testing and Validating Two Morphological Flare Predictors by Logistic Regression Machine Learning
• Testing predictors of eruptivity using parametric flux emergence simulations
• LUCI onboard Lagrange, the Next Generation of EUV Space Weather Monitoring