The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR

Abstract

Current radio interferometers output multi-petabyte-scale volumes of data per year making the storage, transfer, and processing of this data a sizeable challenge. This challenge is expected to grow with the next-generation telescopes such as the Square Kilometre Array. Lossy compression of interferometric data post-correlation can abate this challenge. However, since high-redshift 21-cm studies impose strict precision requirements, the impact of such lossy data compression on the 21-cm signal power spectrum statistic should be understood. We apply Dysco visibility compression, a technique to normalize and quantize specifically designed for radio interferometric data. We establish the level of the compression noise in the power spectrum in comparison to the thermal noise as well as its coherency behavior. Finally, for optimal compression results, we compare the compression noise obtained from different compression settings to a nominal 21-cm signal power. From a single night of observation, we find that the noise introduced due to the compression is more than five orders of magnitude lower than the thermal noise level in the power spectrum. The noise does not affect calibration. The compression noise shows no correlation with the sky signal and has no measurable coherent component. The level of compression error in the power spectrum ultimately depends on the compression settings. Dysco visibility compression is found to be of insignificant concern for 21-cm power spectrum studies. Hence, data volumes can be safely reduced by factors of $\sim 4$ and with insignificant bias to the final power spectrum. Data from SKA-low will likely be compressible by the same factor as LOFAR, owing to the similarities of the two instruments. The same technique can be used to compress data from other telescopes, but a small adjustment of the compression parameters might be required.

12 pages, 9 figures, and 3 tables; submitted to Astronomy and Astrophysics (A&A)

References in corpus (14)

• WSClean: an implementation of a fast, generic wide-field imager for radio astronomy
• The LOFAR Two-metre Sky Survey - I. Survey Description and Preliminary Data Release
• Improved upper limits on the 21-cm signal power spectrum of neutral hydrogen at $\mathbf{z}\approx 9.1$ from LOFAR
• Foreground simulations for the LOFAR - Epoch of Reionization Experiment
• Deep multi-redshift limits on Epoch of Reionisation 21cm Power Spectra from Four Seasons of Murchison Widefield Array Observations
• First Results from HERA Phase I: Upper Limits on the Epoch of Reionization 21 cm Power Spectrum
• Improved Constraints on the 21 cm EoR Power Spectrum and the X-Ray Heating of the IGM with HERA Phase I Observations
• First Season MWA Phase II EoR Power Spectrum Results at Redshift 7
• Distributed Radio Interferometric Calibration
• First upper limits on the 21 cm signal power spectrum from cosmic dawn from one night of observations with NenuFAR
• Compression of interferometric radio-astronomical data
• LOFAR detections of low-frequency radio recombination lines towards Cassiopeia A
• Transient RFI environment of LOFAR-LBA at 72-75 MHz: Impact on ultra-widefield AARTFAAC Cosmic Explorer observations of the redshifted 21-cm signal
• Ionospheric contributions to the excess power in high-redshift 21-cm power-spectrum observations with LOFAR

Cited by in corpus (3)

• Improved upper limits on the 21-cm signal power spectrum at $z=17.0$ and $z=20.3$ from an optimal field observed with NenuFAR
• First upper limits on the 21-cm signal power spectrum of neutral hydrogen at $z=9.16$ from the LOFAR 3C196 field
• Lossless compression of simulated radio interferometric visibilities