A Processing Pipeline for High Volume Pulsar Data Streams
arXiv:1810.06012
Abstract
Pulsar data analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line analysis of data. However modern data acquisition systems are making off-line analyses impractical. They often output multiple simultaneous high volume data streams, significantly increasing data capture rates. This leads to the accumulation of large data volumes, which are prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift with respect to the search for radio pulsars: i) it characterises for the modern era, the key components of a pulsar search science (not signal processing) pipeline, ii) it examines the feasibility of implementing on-line pulsar search via existing tools, iii) problems preventing an easy transition to on-line search are identified and explained, and finally iv) it provides the design for a new prototype pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve pulsar search design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads.
arXiv commentpre-print submitted to Astronomy & Computing
References in corpus (15)
- The ATNF Pulsar Catalogue
- A bright millisecond radio burst of extragalactic origin
- A Fast Radio Burst Host Galaxy
- A real-time fast radio burst: polarization detection and multiwavelength follow-up
- Searching for pulsars using image pattern recognition
- Limits on Fast Radio Bursts at 145 MHz with ARTEMIS, a real-time software backend
- Discovery of 28 pulsars using new techniques for sorting pulsar candidates
- The GMRT High Resolution Southern Sky Survey for pulsars and transients -I. Survey description and initial discoveries
- Real-time, fast radio transient searches with GPU de-dispersion
- A Real-time Coherent Dedispersion Pipeline for the Giant Metrewave Radio Telescope
- PONDER - A Real time software backend for pulsar and IPS observations at the Ooty Radio Telescope
- Hellinger Distance Trees for Imbalanced Streams
- Pulsar Acceleration Searches on the GPU for the Square Kilometre Array
- Matching Radio Catalogs with Realistic Geometry: Application to SWIRE and ATLAS
- New algorithms for radio pulsar search