An investigation on the factors affecting machine learning classifications in $\gamma$-ray astronomy

Abstract

We have investigated a number of factors that can have significant impacts on the classification performance of $\gamma$-ray sources detected by Fermi Large Area Telescope (LAT) with machine learning techniques. We show that a framework of automatic feature selection can construct a simple model with a small set of features which yields better performance over previous results. Secondly, because of the small sample size of the training/test sets of certain classes in $\gamma$-ray, nested re-sampling and cross-validations are suggested for quantifying the statistical fluctuations of the quoted accuracy. We have also constructed a test set by cross-matching the identified active galactic nuclei (AGNs) and the pulsars (PSRs) in the Fermi LAT eight-year point source catalog (4FGL) with those unidentified sources in the previous 3${}^{\mathrm{rd}}$ Fermi LAT Source Catalog (3FGL). Using this cross-matched set, we show that some features used for building classification model with the identified source can suffer from the problem of covariate shift, which can be a result of various observational effects. This can possibly hamper the actual performance when one applies such model in classifying unidentified sources. Using our framework, both AGN/PSR and young pulsar (YNG)/millisecond pulsar (MSP) classifiers are automatically updated with the new features and the enlarged training samples in 4FGL catalog incorporated. Using a two-layer model with these updated classifiers, we have selected 20 promising MSP candidates with confidence scores $>98\%$ from the unidentified sources in 4FGL catalog which can provide inputs for a multi-wavelength identification campaign.

15 pages, 10 figures, 11 tables, accepted for publication in MNRAS

References in corpus (13)

• Fermi Large Area Telescope Second Source Catalog
• Fermi Large Area Telescope First Source Catalog
• A Review of Feature Selection Methods Based on Mutual Information
• Model-Agnostic Interpretability of Machine Learning
• Fermi Large Area Telescope Bright Gamma-ray Source List
• Data Mining and Machine Learning in Astronomy
• A Statistical Approach to Recognizing Source Classes for Unassociated Sources in the First Fermi-LAT Catalog
• Classification and Ranking of Fermi LAT Gamma-ray Sources from the 3FGL Catalog using Machine Learning Techniques
• Fermi's Sibyl: Mining the gamma-ray sky for dark matter subhaloes
• Machine Learning Classification of SDSS Transient Survey Images
• A Machine Learning Method to Infer Fundamental Stellar Parameters from Photometric Light Curves
• Murchison Widefield Array Limits on Radio Emission from ANTARES Neutrino Events
• Searches for millisecond pulsar candidates among the unidentified Fermi objects

Cited by in corpus (19)

• The Third Fermi Large Area Telescope Catalog of Gamma-ray Pulsars
• The TRAPUM L-band survey for pulsars in Fermi-LAT gamma-ray sources
• Discovery of a New Redback Millisecond Pulsar Candidate: 4FGL J0940.3-7610
• Classification of Fermi-LAT sources with deep learning using energy and time spectra
• SKA Science Data Challenge 2: analysis and results
• Machine learning methods for constructing probabilistic Fermi-LAT catalogs
• Searching for TeV Candidates in 4LAC High-synchrotron-peaked Frequency BL Lac Objects
• Revealing a New Black Widow Binary 4FGL J0336.0+7502
• Multi-class classification of Fermi-LAT sources with hierarchical class definition
• A Multi-wavelength search for Black Widows and Redbacks counterparts of candidate $\gamma$-ray millisecond pulsars
• Searches for Pulsar-like Candidates from Unidentified Objects in the Third Catalog of Hard Fermi-LAT (3FHL) sources with Machine Learning Techniques
• Towards resolving the Galactic center GeV excess with millisecond-pulsar-like sources using machine learning
• A New Mid-Infrared and X-ray Machine Learning Algorithm to Discover Compton-thick AGN
• Effect of covariate shift on multi-class classification of Fermi-LAT sources
• Searching for dark matter subhalos in the Fermi-LAT catalog with Bayesian neural networks
• Drifting Features: Detection and evaluation in the context of automatic RRLs identification in VVV
• Feature Selection Techniques for CR Isotope Identification with the AMS-02 Experiment in Space
• Classification of Fermi-LAT blazars with Bayesian neural networks
• Exploring the nature of Galactic unassociated sources detected by the Fermi-LAT