Effect of covariate shift on multi-class classification of Fermi-LAT sources

Abstract

Probabilistic classification of unassociated Fermi-LAT sources using machine learning methods has an implicit assumption that the distributions of associated and unassociated sources are the same as a function of source parameters, which is not the case for the Fermi-LAT catalogs. The problem of different distributions of training and testing (or target) datasets as a function of input features (covariates) is known as the covariate shift. In this paper, we, for the first time, quantitatively estimate the effect of the covariate shift on the multi-class classification of Fermi-LAT sources. We introduce sample weights proportional to the ratio of unassociated to associated source probability density functions so that associated sources in areas, which are densely populated with unassociated sources, have more weight than the sources in areas with few unassociated sources. We find that the covariate shift has relatively little effect on the predicted probabilities, i.e., the training can be performed either with weighted or with unweighted samples, which is generally expected for the covariate shift problems. The main effect of the covariate shift is on the estimated performance of the classification. Depending on the class, the covariate shift can lead up to 10 - 20% reduction in precision and recall compared to the estimates, where the covariate shift is not taken into account.

arXiv comment16 pages, 19 figures, results are available online at https://zenodo.org/doi/10.5281/zenodo.8140548, v2: update for the 4FGL-DR4 catalog, v3: published in RASTI

References in corpus (12)

• Astropy: A Community Python Package for Astronomy
• Fermi Large Area Telescope Fourth Source Catalog
• Incremental Fermi Large Area Telescope Fourth Source Catalog
• A Statistical Approach to Recognizing Source Classes for Unassociated Sources in the First Fermi-LAT Catalog
• Classification and Ranking of Fermi LAT Gamma-ray Sources from the 3FGL Catalog using Machine Learning Techniques
• 3FGL Demographics Outside the Galactic Plane using Supervised Machine Learning: Pulsar and Dark Matter Subhalo Interpretations
• Research and characterisation of blazar candidates among the Fermi/LAT 3FGL catalogue using multivariate classifications
• An investigation on the factors affecting machine learning classifications in $\gamma$-ray astronomy
• Classification of Fermi-LAT sources with deep learning using energy and time spectra
• Searching for AGN and Pulsar Candidates in 4FGL Unassociated Sources Using Machine Learning
• Machine learning methods for constructing probabilistic Fermi-LAT catalogs
• Multi-class classification of Fermi-LAT sources with hierarchical class definition

Cited by in corpus (2)

• Towards resolving the Galactic center GeV excess with millisecond-pulsar-like sources using machine learning
• Exploring the nature of Galactic unassociated sources detected by the Fermi-LAT