Pipeline Collector: gathering performance data for distributed astronomical pipelines

Abstract

Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We tested the tool by running the pipeline on several different compute platforms and collected the performance data. Based on this data, we make well informed recommendations on future hardware and software upgrades. The aim of these upgrades is to accelerate the slowest processing steps for this LOFAR pipeline. The pipeline collector suite is open source and will be incorporated in future LOFAR pipelines to create a performance database for all LOFAR processing.

arXiv comment15 pages, 14 figures, Astronomy and Computing, 2018, ISSN 2213-1337

References in corpus (11)

• LOFAR: The LOw-Frequency ARray
• The Murchison Widefield Array: the Square Kilometre Array Precursor at low radio frequencies
• The LOFAR Two-metre Sky Survey - I. Survey Description and Preliminary Data Release
• The Murchison Widefield Array: Design Overview
• LOFAR facet calibration
• LOFAR 150-MHz observations of the Boötes field: Catalogue and Source Counts
• Radio interferometric gain calibration as a complex optimization problem
• The Helioseismic and Magnetic Imager (HMI) Vector Magnetic Field Pipeline: Optimization of the Spectral Line Inversion Code
• The LOFAR radio environment
• Cobalt: A GPU-based correlator and beamformer for LOFAR
• Discovery of Carbon Radio Recombination Lines in absorption towards Cygnus~A

Cited by in corpus (4)

• Extremely deep 150 MHz source counts from the LoTSS Deep Fields
• Revived Fossil Plasma Sources in Galaxy Clusters
• The first detection of radio recombination lines at cosmological distances
• Searching for the largest bound atoms in space