Quantifying Uncertainty in Deep Learning Approaches to Radio Galaxy Classification

Abstract

In this work we use variational inference to quantify the degree of uncertainty in deep learning model predictions of radio galaxy classification. We show that the level of model posterior variance for individual test samples is correlated with human uncertainty when labelling radio galaxies. We explore the model performance and uncertainty calibration for different weight priors and suggest that a sparse prior produces more well-calibrated uncertainty estimates. Using the posterior distributions for individual weights, we demonstrate that we can prune 30% of the fully-connected layer weights without significant loss of performance by removing the weights with the lowest signal-to-noise ratio. A larger degree of pruning can be achieved using a Fisher information based ranking, but both pruning methods affect the uncertainty calibration for Fanaroff-Riley type I and type II radio galaxies differently. Like other work in this field, we experience a cold posterior effect, whereby the posterior must be down-weighted to achieve good predictive performance. We examine whether adapting the cost function to accommodate model misspecification can compensate for this effect, but find that it does not make a significant difference. We also examine the effect of principled data augmentation and find that this improves upon the baseline but also does not compensate for the observed effect. We interpret this as the cold posterior effect being due to the overly effective curation of our training sample leading to likelihood misspecification, and raise this as a potential issue for Bayesian deep learning approaches to radio galaxy classification in future.

arXiv commentaccepted by MNRAS

References in corpus (21)

• The Seventh Data Release of the Sloan Digital Sky Survey
• LOFAR: The LOw-Frequency ARray
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• On the fundamental dichotomy in the local radio-AGN population: accretion, evolution, and host galaxy properties
• Science with ASKAP - the Australian Square Kilometre Array Pathfinder
• Radio galaxies with a `double-double' morphology: I - Analysis of the radio properties and evidence for interrupted activity in active galactic nuclei
• Revisiting the Fanaroff-Riley dichotomy and radio-galaxy morphology with the LOFAR Two-Metre Sky Survey (LoTSS)
• Relativistic Jets and the Fanaroff-Riley Classification of Radio Galaxies
• Classifying Radio Galaxies with Convolutional Neural Network
• Luminosity function, sizes and FR dichotomy of radio-loud AGN
• Radio Galaxy Zoo: host galaxies and radio morphologies derived from visual inspection
• Particle content, radio-galaxy morphology and jet power: all radio-loud AGN are not equal
• Radio Galaxy Zoo: ClaRAN - A Deep Learning Classifier for Radio Morphologies
• Radio Galaxy Zoo: Compact and extended radio source classification with deep learning
• The nuclear properties and extended morphologies of powerful radio galaxies: the roles of host galaxy and environment
• Fanaroff-Riley classification of radio galaxies using group-equivariant convolutional neural networks
• Attention-gating for improved radio galaxy classification
• Transfer learning for radio galaxy classification
• Science with the Murchison Widefield Array: Phase I Results and Phase II Opportunities
• Rapid sorting of radio galaxy morphology using Haralick features
• Structured Variational Inference for Simulating Populations of Radio Galaxies

Cited by in corpus (6)

• Optimal, fast, and robust inference of reionization-era cosmology with the 21cmPIE-INN
• Using Bayesian Deep Learning to infer Planet Mass from Gaps in Protoplanetary Disks
• Finding AGN remnant candidates based on radio morphology with machine learning
• Probabilistic learning for pulsar classification
• Radio Galaxy Zoo: Using semi-supervised learning to leverage large unlabelled data-sets for radio galaxy classification under data-set shift
• Radio Galaxy Zoo: Morphological classification by Fanaroff-Riley designation using self-supervised pre-training