How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events

Abstract

We present a case study of solar flare forecasting by means of metadata feature time series, by treating it as a prominent class-imbalance and temporally coherent problem. Taking full advantage of pre-flare time series in solar active regions is made possible via the Space Weather Analytics for Solar Flares (SWAN-SF) benchmark dataset; a partitioned collection of multivariate time series of active region properties comprising 4075 regions and spanning over 9 years of the Solar Dynamics Observatory (SDO) period of operations. We showcase the general concept of temporal coherence triggered by the demand of continuity in time series forecasting and show that lack of proper understanding of this effect may spuriously enhance models' performance. We further address another well-known challenge in rare event prediction, namely, the class-imbalance issue. The SWAN-SF is an appropriate dataset for this, with a 60:1 imbalance ratio for GOES M- and X-class flares and a 800:1 for X-class flares against flare-quiet instances. We revisit the main remedies for these challenges and present several experiments to illustrate the exact impact that each of these remedies may have on performance. Moreover, we acknowledge that some basic data manipulation tasks such as data normalization and cross validation may also impact the performance -- we discuss these problems as well. In this framework we also review the primary advantages and disadvantages of using true skill statistic and Heidke skill score, as two widely used performance verification metrics for the flare forecasting task. In conclusion, we show and advocate for the benefits of time series vs. point-in-time forecasting, provided that the above challenges are measurably and quantitatively addressed.

arXiv commentThe Astrophysical Journal Supplement series (2021)

References in corpus (26)

• SMOTE: Synthetic Minority Over-sampling Technique
• The Solar Optical Telescope for the Hinode Mission: An Overview
• An Observational Overview of Solar Flares
• The Helioseismic and Magnetic Imager (HMI) Vector Magnetic Field Pipeline: SHARPs -- Space-weather HMI Active Region Patches
• The Helioseismic and Magnetic Imager (HMI) Vector Magnetic Field Pipeline: Overview and Performance
• The Challenge of Machine Learning in Space Weather Nowcasting and Forecasting
• Solar Flare Prediction Using SDO/HMI Vector Magnetic Field Data with a Machine-Learning Algorithm
• Toward Reliable Benchmarking of Solar Flare Forecasting Methods
• 25 Years of Self-Organized Criticality: Solar and Astrophysics
• The Polarimetric and Helioseismic Imager on Solar Orbiter
• A Comparison of Flare Forecasting Methods, I: Results from the "All-Clear" Workshop
• Solar Flare Prediction Model with Three Machine-Learning Algorithms Using Ultraviolet Brightening and Vector Magnetogram
• Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning
• Automated Solar Flare Statistics in Soft X-rays over 37 Years of GOES Observations - The Invariance of Self-Organized Criticality during Three Solar Cycles
• A Comparison of Flare Forecasting Methods. II. Benchmarks, Metrics and Performance Results for Operational Solar Flare Forecasting Systems
• What is the relationship between photospheric flow fields and solar flares?
• A hybrid supervised/unsupervised machine learning approach to solar flare prediction
• The importance of ensemble techniques for operational space weather forecasting
• Are Solar Active Regions with Major Flares More Fractal, Multifractal, or Turbulent than Others?
• A Comparison of Flare Forecasting Methods. III. Systematic Behaviors of Operational Solar Flare Forecasting Systems
• Feature ranking of active region source properties in solar flare forecasting and the uncompromised stochasticity of flare occurrence
• Ensemble Forecasting of Major Solar Flares -- First Results
• A Comparison of Flare Forecasting Methods. IV. Evaluating Consecutive-Day Forecasting Patterns
• A Framework for Designing and Evaluating Solar Flare Forecasting Systems
• Solar Flares Forecasting Using Time Series and Extreme Gradient Boosting Ensembles
• Ensemble Forecasting of Major Solar Flares: Methods for Combining Models

Cited by in corpus (10)

• Predicting Solar Flares Using CNN and LSTM on Two Solar Cycles of Active Region Data
• Implementation paradigm for supervised flare forecasting studies: a deep learning application with video data
• Towards Coupling Full-disk and Active Region-based Flare Prediction for Operational Space Weather Forecasting
• Explaining Full-disk Deep Learning Model for Solar Flare Prediction using Attribution Methods
• Explainable Deep Learning-based Solar Flare Prediction with post hoc Attention for Operational Forecasting
• Deep learning reconstruction of sunspot vector magnetic fields for forecasting solar storms
• On the differences in the periodic behaviour of magnetic helicity flux in flaring active regions with and without X-class events
• Using Multiple Instance Learning for Explainable Solar Flare Prediction
• The Random Hivemind: An Ensemble Deep Learner Application to Solar Energetic Particle Prediction Problem
• A Comparative Analysis of Machine-learning Models for Solar Flare Forecasting: Identifying High-performing Active Region Flare Indicators