Scalability Model for the LOFAR Direction Independent Pipeline
arXiv:1906.11516 · doi:10.1016/j.ascom.2019.100293
Abstract
LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the performance of this pipeline is important when trying to optimize the throughput for large projects, such as LoTSS and other deep surveys. Making a model of its completion time will enable us to predict the time taken to process large data sets, optimize our parameter choices, help schedule other LOFAR processing services, and predict processing time for future large radio telescopes. We tested the prefactor pipeline by scaling several parameters, notably number of CPUs, data size and size of calibration sky model. We present these results as a comprehensive model which will be used to predict processing time for a wide range of processing parameters. We also discover that smaller calibration models lead to significantly faster calibration times, while the calibration results do not significantly degrade in quality. Finally, we validate the model and compare predictions with production runs from the past six months, quantifying the performance penalties incurred by processing on a shared cluster. We conclude by noting the utility of the results and model for the LoTSS Survey, LOFAR as a whole and for other telescopes.
arXiv comment17 pages, 24 figures
References in corpus (12)
- LOFAR: The LOw-Frequency ARray
- WSClean: an implementation of a fast, generic wide-field imager for radio astronomy
- The GMRT 150 MHz All-sky Radio Survey: First Alternative Data Release TGSS ADR1
- The LOFAR Two-metre Sky Survey - I. Survey Description and Preliminary Data Release
- The LOFAR Two-metre Sky Survey - II. First data release
- LOFAR facet calibration
- LOFAR 150-MHz observations of the Boötes field: Catalogue and Source Counts
- Faceting for direction-dependent spectral deconvolution
- Radio interferometric gain calibration as a complex optimization problem
- Radio Interferometric Calibration Using The SAGE Algorithm
- Image formation in synthetic aperture radio telescopes
- Predictive Performance Modeling for Distributed Computing using Black-Box Monitoring and Machine Learning