Fifty Years of Pulsar Candidate Selection: From simple filters to a new principled real-time classification approach

Abstract

Improving survey specifications are causing an exponential rise in pulsar candidate numbers and data volumes. We study the candidate filters used to mitigate these problems during the past fifty years. We find that some existing methods such as applying constraints on the total number of candidates collected per observation, may have detrimental effects on the success of pulsar searches. Those methods immune to such effects are found to be ill-equipped to deal with the problems associated with increasing data volumes and candidate numbers, motivating the development of new approaches. We therefore present a new method designed for on-line operation. It selects promising candidates using a purpose-built tree-based machine learning classifier, the Gaussian Hellinger Very Fast Decision Tree (GH-VFDT), and a new set of features for describing candidates. The features have been chosen so as to i) maximise the separation between candidates arising from noise and those of probable astrophysical origin, and ii) be as survey-independent as possible. Using these features our new approach can process millions of candidates in seconds (~1 million every 15 seconds), with high levels of pulsar recall (90%+). This technique is therefore applicable to the large volumes of data expected to be produced by the Square Kilometre Array (SKA). Use of this approach has assisted in the discovery of 20 new pulsars in data obtained during the LOFAR Tied-Array All-Sky Survey (LOTAAS).

arXiv commentAccepted for publication in MNRAS, 20 pages, 8 figures. See http://www.jb.man.ac.uk/pulsar/Surveys.html for survey data, and https://dx.doi.org/10.6084/m9.figshare.3080389.v1 for our data

References in corpus (50)

• The ATNF Pulsar Catalogue
• A bright millisecond radio burst of extragalactic origin
• The ATNF Pulsar Catalogue
• A Population of Fast Radio Bursts at Cosmological Distances
• How small were the first cosmological objects?
• The Parkes Multibeam Pulsar Survey: I. Observing and Data Analysis Systems, Discovery and Timing of 100 Pulsars
• Fast Radio Burst Discovered in the Arecibo Pulsar ALFA Survey
• The High Time Resolution Universe Pulsar Survey I: System configuration and initial discoveries
• Detection of 16 Gamma-Ray Pulsars Through Blind Frequency Searches Using the Fermi LAT
• A real-time fast radio burst: polarization detection and multiwavelength follow-up
• Arecibo Pulsar Survey Using ALFA. I. Survey Strategy and First Discoveries
• Discovery, Progenitor & Early Evolution of a Stripped Envelope Supernova iPTF13bvn
• On the origin of a highly-dispersed coherent radio burst
• The Green Bank Northern Celestial Cap Pulsar Survey - I: Survey Description, Data Analysis, and Initial Results
• Pulsar searches and timing with the SKA
• Three Millisecond Pulsars in FERMI LAT Unassociated Bright Sources
• The Parkes Multibeam Pulsar Survey: V. Finding binary and millisecond pulsars
• The Swinburne Intermediate Latitude Pulsar Survey
• Searching for pulsars using image pattern recognition
• Arecibo Pulsar Survey Using ALFA: Probing Radio Pulsar Intermittency and Transients
• Selection of radio pulsar candidates using artificial neural networks
• The Northern High Time Resolution Universe Pulsar Survey I: Setup and initial discoveries
• The Green Bank Telescope 350 MHz Drift-scan Survey I: Survey Observations and the Discovery of 13 Pulsars
• A 1.4-GHz Arecibo Survey for Pulsars in Globular Clusters
• Discovery of 14 radio pulsars in a survey of the Magellanic Clouds
• The LOFAR Pilot Surveys for Pulsars and Fast Radio Transients
• Limits on Fast Radio Bursts at 145 MHz with ARTEMIS, a real-time software backend
• Discovery of 28 pulsars using new techniques for sorting pulsar candidates
• SPINN: a straightforward machine learning solution to the pulsar candidate selection problem
• Goals, Strategies and First Discoveries of AO327, the Arecibo All-Sky 327 MHz Drift Pulsar Survey
• The High Time Resolution Universe Survey VI: An Artificial Neural Network and Timing of 75 Pulsars
• Pulsar science with the Five hundred metre Aperture Spherical Telescope
• PSR J1930-1852: a pulsar in the widest known orbit around another neutron star
• A Survey of 56 Mid-latitude EGRET Error Boxes for Radio Pulsars
• A 6.5-GHz Multibeam Pulsar Survey
• PEACE: Pulsar Evaluation Algorithm for Candidate Extraction -- A software package for post-analysis processing of pulsar survey candidates
• The GMRT High Resolution Southern Sky Survey for pulsars and transients -I. Survey description and initial discoveries
• Long term variation of the solar diurnal anisotropy of galactic cosmic rays observed with the Nagoya multi-directional muon detector
• The Parkes multibeam pulsar survey: VII. Timing of four millisecond pulsars and the underlying spin period distribution of the Galactic millisecond pulsar population
• A coherent acceleration search of the Parkes multi-beam pulsar survey - techniques and the discovery and timing of 16 pulsars
• The Pulsar Search Collaboratory: Discovery and Timing of Five New Pulsars
• Discovery and timing of the first 8gr8 Cygnus survey pulsars
• Timing of pulsars found in a deep Parkes multibeam survey
• The Pulsar Search Collaboratory
• The Arecibo 430-MHz Intermediate Galactic Latitude Survey: Discovery of Nine Radio Pulsars
• A survey for pulsars in EGRET error boxes
• Searching for Millisecond Pulsars: Surveys, Techniques and Prospects
• A Study on Classification in Imbalanced and Partially-Labelled Data Streams
• Comparison of RFI Mitigation Strategies for Dispersed Pulse Detection
• SPAN512: A new mid-latitude pulsar survey with the Nancay Radio Telescope

Cited by in corpus (52)

• The FAST Galactic Plane Pulsar Snapshot survey: I. Project design and pulsar discoveries
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• A Hybrid Ensemble method for Pulsar Candidate Classification
• The LOFAR Tied-Array All-Sky Survey (LOTAAS): Survey overview and initial pulsar discoveries
• Optimal periodicity searching: Revisiting the Fast Folding Algorithm for large-scale pulsar surveys
• Random Partitioning Forest for Point-Wise and Collective Anomaly Detection -- Application to Intrusion Detection
• The High Time Resolution Universe survey XIV: Discovery of 23 pulsars through GPU-accelerated reprocessing
• Applying Deep Learning to Fast Radio Burst Classification
• Separation of pulsar signals from noise with supervised machine learning algorithms
• Fractional norms and quasinorms do not help to overcome the curse of dimensionality
• Single-pulse classifier for the LOFAR Tied-Array All-sky Survey
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• An in-depth investigation of 11 pulsars discovered by FAST
• Concat Convolutional Neural Network for Pulsar Candidate Selection
• Comparing Multi-class, Binary and Hierarchical Machine Learning Classification schemes for variable stars
• Pulsar Candidate Identification Using Semi-Supervised Generative Adversarial Networks
• Pushchino multibeam pulsar search: I. Targeted search of weak pulsars
• Improved Weighted Random Forest for Classification Problems
• Pulsar Searches with the SKA
• A Framework for Assessing the Performance of Pulsar Search Pipelines
• Searching for pulsars associated with polarised point sources using LOFAR: Initial discoveries from the TULIPP project
• Discovery of 37 new pulsars through GPU-accelerated reprocessing of archival data of the Parkes Multibeam Pulsar Survey
• Pulsars Detection by Machine Learning with Very Few Features
• Active deep learning method for the discovery of objects of interest in large spectroscopic surveys
• PulsarX: a new pulsar searching package -I. A high performance folding program for pulsar surveys
• The Southern-sky MWA Rapid Two-metre (SMART) pulsar survey -- I. Survey design and processing pipeline
• A LOFAR radio search for single and periodic pulses from M31
• Pulsar Candidate Sifting Using Multi-input Convolution Neural Networks
• Online Neural Networks for Change-Point Detection
• Pulsar Classification: Comparing Quantum Convolutional Neural Networks and Quantum Support Vector Machines
• The TRAPUM Small Magellanic Cloud pulsar survey with MeerKAT: I. Discovery of seven new pulsars and two Pulsar Wind Nebula associations
• The UTMOST Survey for Magnetars, Intermittent pulsars, RRATs and FRBs I: System description and overview
• Fourier-domain dedispersion
• Classical Ensembles of Single-Qubit Quantum Variational Circuits for Classification
• Topology of deep neural networks
• PaloBoost: An Overfitting-robust TreeBoost with Out-of-Bag Sample Regularization Techniques
• Machine Learning Pipeline for Pulsar Star Dataset
• Wasserstein Measure Coresets
• Detecting Pulsars with Neural Networks: A Proof of Concept
• Dealing with the data imbalance problem on pulsar candidates sifting based on feature selection
• Investigation of a Machine learning methodology for the SKA pulsar search pipeline
• Enabling pulsar and fast transient searches using coherent dedispersion
• A Method for Pulsar Searching: Combining a Two-dimensional Autocorrelation Profile Map and a Deep Convolutional Neural Network
• Quantifying With Only Positive Training Data
• Fast Statistical Leverage Score Approximation in Kernel Ridge Regression
• More Powerful Selective Kernel Tests for Feature Selection
• Particle Filtering Methods for Stochastic Optimization with Application to Large-Scale Empirical Risk Minimization
• Predictive Power of Nearest Neighbors Algorithm under Random Perturbation
• Distributed Nearest Neighbor Classification
• Fifty Years of Candidate Pulsar Selection - What next?
• The GMRT High Resolution Southern Sky Survey for pulsars and transients -- III: searching for long period pulsars
• Searching for pulsars with phase characteristics