Toward Reliable Benchmarking of Solar Flare Forecasting Methods

Abstract

Solar flares occur in complex sunspot groups, but it remains unclear how the probability of producing a flare of a given magnitude relates to the characteristics of the sunspot group. Here, we use Geostationary Operational Environmental Satellite X-ray flares and McIntosh group classifications from solar cycles 21 and 22 to calculate average flare rates for each McIntosh class and use these to determine Poisson probabilities for different flare magnitudes. Forecast verification measures are studied to find optimum thresholds to convert Poisson flare probabilities into yes/no predictions of cycle 23 flares. A case is presented to adopt the true skill statistic (TSS) as a standard for forecast comparison over the commonly used Heidke skill score (HSS). In predicting flares over 24 hr, the maximum values of TSS achieved are 0.44 (C-class), 0.53 (M-class), 0.74 (X-class), 0.54 (>=M1.0), and 0.46 (>=C1.0). The maximum values of HSS are 0.38 (C-class), 0.27 (M-class), 0.14 (X-class), 0.28 (>=M1.0), and 0.41 (>=C1.0). These show that Poisson probabilities perform comparably to some more complex prediction systems, but the overall inaccuracy highlights the problem with using average values to represent flaring rate distributions.

7 pages, 1 figure

References in corpus (2)

• A Statistical Solar Flare Forecast Method
• Reconciliation of Waiting Time Statistics of Solar Flares Observed in Hard X-Rays

Cited by in corpus (84)

• The Challenge of Machine Learning in Space Weather Nowcasting and Forecasting
• Solar Flare Prediction Using SDO/HMI Vector Magnetic Field Data with a Machine-Learning Algorithm
• The origin, early evolution and predictability of solar eruptions
• A Comparison of Flare Forecasting Methods, I: Results from the "All-Clear" Workshop
• Solar Flare Prediction Model with Three Machine-Learning Algorithms Using Ultraviolet Brightening and Vector Magnetogram
• Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning
• Deep Flare Net (DeFN) model for solar flare prediction
• Forecasting the Arrival Time of Coronal Mass Ejections: Analysis of the CCMC CME Scoreboard
• Predicting Solar Flares Using a Long Short-Term Memory Network
• Predicting Solar Flares Using SDO/HMI Vector Magnetic Data Product and Random Forest Algorithm
• Predicting Coronal Mass Ejections Using Machine Learning Methods
• A Comparison of Flare Forecasting Methods. II. Benchmarks, Metrics and Performance Results for Operational Solar Flare Forecasting Systems
• Modeling observations of solar coronal mass ejections with heliospheric imagers verified with the Heliophysics System Observatory
• Flare Prediction Using Photospheric and Coronal Image Data
• A hybrid supervised/unsupervised machine learning approach to solar flare prediction
• Predicting Solar Flares Using CNN and LSTM on Two Solar Cycles of Active Region Data
• How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events
• A Comparison of Flare Forecasting Methods. III. Systematic Behaviors of Operational Solar Flare Forecasting Systems
• Feature ranking of active region source properties in solar flare forecasting and the uncompromised stochasticity of flare occurrence
• Solar Flare Intensity Prediction with Machine Learning Models
• Very Long-period Pulsations before the Onset of Solar Flares
• Prediction of Solar Flares Using Unique Signatures of Magnetic Field Images
• Structure and Evolution of Magnetic Fields Associated with Solar Eruptions (Invited Review)
• Ensemble Forecasting of Major Solar Flares -- First Results
• A Comparison of Flare Forecasting Methods. IV. Evaluating Consecutive-Day Forecasting Patterns
• UFCORIN: A Fully Automated Predictor of Solar Flares in GOES X-Ray Flux
• Flaring Rates and the Evolution of Sunspot Group McIntosh Classifications
• Prediction of Solar Flare Size and Time-to-Flare Using Support Vector Machine Regression
• A blueprint of state-of-the-art techniques for detecting quasi-periodic pulsations in solar and stellar flares
• Operational solar flare prediction model using Deep Flare Net
• An automated classification approach to ranking photospheric proxies of magnetic energy build-up
• On flare predictability based on sunspot group evolution
• Real-time flare prediction based on distinctions between flaring and non-flaring active region spectra
• Statistical study of magnetic non-potential measures in confined and eruptive flares
• Verification of operational solar flare forecast: Case of Regional Warning Center Japan
• On the generation of probabilistic forecasts from deterministic models
• Predicting Coronal Mass Ejections Using SDO/HMI Vector Magnetic Data Products and Recurrent Neural Networks
• Implementation paradigm for supervised flare forecasting studies: a deep learning application with video data
• Connecting Coronal Mass Ejections to their Solar Active Region Sources: Combining Results from the HELCATS and FLARECAST Projects
• Forecasting the Ambient Solar Wind with Numerical Models. II. An Adaptive Prediction System for Specifying Solar Wind Speed Near the Sun
• Self-Organized Criticality in Solar and Stellar Flares: Are Extreme Events Scale-Free ?
• Fine-grained Solar Flare Forecasting Based on the Hybrid Convolutional Neural Networks
• Predictive Capabilities of Avalanche Models for Solar Flares
• Reliable Probability Forecast of Solar Flares: Deep Flare Net-Reliable (DeFN-R)
• Solar Flares Complex Networks
• Toward a Next Generation Particle Precipitation Model: Mesoscale Prediction Through Machine Learning (a Case Study and Framework for Progress)
• Using gradient boosting regression to improve ambient solar wind model predictions
• Magnetic Nonpotentiality in Photospheric Active Regions as a Predictor of Solar Flares
• Using the Maximum X-ray Flux Ratio and X-ray Background to Predict Solar Flare Class
• A Framework for Designing and Evaluating Solar Flare Forecasting Systems
• Machine learning for predicting the Bz magnetic field component from upstream in situ observations of solar coronal mass ejections
• Identifying preflare spectral features using explainable artificial intelligence
• DeepSun: Machine-Learning-as-a-Service for Solar Flare Prediction
• Solar Flare Index Prediction Using SDO/HMI Vector Magnetic Data Products with Statistical and Machine Learning Methods
• Distinguishing between Flaring and Non-Flaring Active Regions: A Machine Learning Perspective
• Solar Flares Forecasting Using Time Series and Extreme Gradient Boosting Ensembles
• Properties of Flare-Imminent versus Flare-Quiet Active Regions from the Chromosphere through the Corona II: NonParametric Discriminant Analysis Results from the NWRA Classification Infrastructure (NCI)
• Forecasting GICs and geoelectric fields from solar wind data using LSTMs: application in Austria
• The characteristics of flare- and CME-productive solar active regions
• Ensemble Forecasting of Major Solar Flares: Methods for Combining Models
• Prediction of Dst during solar minimum using in situ measurements at L5
• Predicting Solar Energetic Particles Using SDO/HMI Vector Magnetic Data Products and a Bidirectional LSTM Network
• Automatic detection of small-scale EUV brightenings observed by the Solar Orbiter/EUI
• Machine learning as a flaring storm warning machine: Was a warning machine for the September 2017 solar flaring storm possible?
• Performance of Major Flare Watches from the Max Millennium Program (2001-2010)
• Automatic Detection of Magnetic delta in Sunspot Groups
• Global Energetics of Solar Flares. IX. Refined Magnetic Modeling
• On the possibility to probe the flare productivity of an active region on the early stage of emergence
• Synchronization of Small-scale Magnetic Features, Blinkers, and Coronal Bright Points
• One-dimensional Convolutional Neural Networks for Detecting Transiting Exoplanets
• Solar Polar Flux Redistribution based on Observed Coronal Holes
• Real-time solar image classification: assessing spectral, pixel-based approaches
• Solar extreme events
• On the evolution of a sub-C class flare: a showcase for the capabilities of the revamped Catania Solar Telescope
• Frozen-in Fractals All Around: Inferring the Large Scale Effects of Small-Scale Magnetic Structure
• Classifying Signatures of Sudden Ionospheric Disturbances
• Space weather: the solar perspective -- an update to Schwenn (2006)
• On the non-Kolmogorov nature of flare-productive solar active regions
• Solar Flare Prediction Using Long Short-term Memory (LSTM) and Decomposition-LSTM with Sliding Window Pattern Recognition
• Evaluating Pointing Strategies for Future Solar Flare Missions
• Bypassing the static input size of neural networks in flare forecasting by using spatial pyramid pooling
• A Comparative Analysis of Machine-learning Models for Solar Flare Forecasting: Identifying High-performing Active Region Flare Indicators
• ALEXIS: Recreating the X-Ray Emission from the Full Solar Disk as a Linear Combination of Discrete Regions in the Extreme Ultraviolet and Soft X-Rays
• Image patch analysis of sunspots and active regions. II. Clustering via matrix factorization