Validity in machine learning for extreme event attribution

Abstract

Extreme event attribution (EEA), which assesses the extent to which disasters are caused by climate change, is crucial for informing climate policy and legal proceedings. Machine learning is increasingly used for EEA by modeling rare weather events too complex or computationally intensive for traditional methods. However, its validity remains unclear, as machine learning applications are criticized for bias and lack of robustness. Here we evaluate machine learning for EEA using California wildfire data from 2003-2020. We identify three major threats to validity: (1) individual event attribution estimates are highly sensitive to algorithmic design choices; (2) common performance metrics like Brier score are not strongly correlated with attribution error, facilitating suboptimal model selection; and (3) distribution shift across climate scenarios substantially degrades predictive performance. We propose a more robust attribution analysis using aggregate estimates and additional evaluation metrics for predictive performance and distribution shift.