The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR

Abstract

Current radio interferometers output multi-petabyte-scale volumes of data per year making the storage, transfer, and processing of this data a sizeable challenge. This challenge is expected to grow with the next-generation telescopes such as the Square Kilometre Array. Lossy compression of interferometric data post-correlation can abate this challenge. However, since high-redshift 21-cm studies impose strict precision requirements, the impact of such lossy data compression on the 21-cm signal power spectrum statistic should be understood. We apply Dysco visibility compression, a technique to normalize and quantize specifically designed for radio interferometric data. We establish the level of the compression noise in the power spectrum in comparison to the thermal noise as well as its coherency behavior. Finally, for optimal compression results, we compare the compression noise obtained from different compression settings to a nominal 21-cm signal power. From a single night of observation, we find that the noise introduced due to the compression is more than five orders of magnitude lower than the thermal noise level in the power spectrum. The noise does not affect calibration. The compression noise shows no correlation with the sky signal and has no measurable coherent component. The level of compression error in the power spectrum ultimately depends on the compression settings. Dysco visibility compression is found to be of insignificant concern for 21-cm power spectrum studies. Hence, data volumes can be safely reduced by factors of $\sim 4$ and with insignificant bias to the final power spectrum. Data from SKA-low will likely be compressible by the same factor as LOFAR, owing to the similarities of the two instruments. The same technique can be used to compress data from other telescopes, but a small adjustment of the compression parameters might be required.

arXiv comment12 pages, 9 figures, and 3 tables; submitted to Astronomy and Astrophysics (A&A)

References in corpus (32)

• LOFAR: The LOw-Frequency ARray
• The Murchison Widefield Array: the Square Kilometre Array Precursor at low radio frequencies
• WSClean: an implementation of a fast, generic wide-field imager for radio astronomy
• Hydrogen Epoch of Reionization Array (HERA)
• The LOFAR Two-metre Sky Survey - I. Survey Description and Preliminary Data Release
• Cosmological Parameter Estimation Using 21 cm Radiation from the Epoch of Reionization
• A morphological algorithm for improving radio-frequency interference detection
• Reionization and the Cosmic Dawn with the Square Kilometre Array
• A simulation calibrated limit on the HI power spectrum from the GMRT Epoch of Reionization experiment
• Improved upper limits on the 21-cm signal power spectrum of neutral hydrogen at $\mathbf{z}\approx 9.1$ from LOFAR
• Upper limits on the 21-cm Epoch of Reionization power spectrum from one night with LOFAR
• Foreground simulations for the LOFAR - Epoch of Reionization Experiment
• Toward Epoch of Reionization Measurements with Wide-Field Radio Observations
• Deep multi-redshift limits on Epoch of Reionisation 21cm Power Spectra from Four Seasons of Murchison Widefield Array Observations
• First Results from HERA Phase I: Upper Limits on the Epoch of Reionization 21 cm Power Spectrum
• Calibration Requirements for Detecting the 21 cm Epoch of Reionization Power Spectrum and Implications for the SKA
• Improved Constraints on the 21 cm EoR Power Spectrum and the X-Ray Heating of the IGM with HERA Phase I Observations
• Initial deep LOFAR observations of Epoch of Reionization windows: I. The North Celestial Pole
• A simplified, lossless re-analysis of PAPER-64
• Realistic Simulations of the Galactic Polarized Foreground: Consequences for 21-cm Reionization Detection Experiments
• The Evolution of 21-cm Structure (EOS): public, large-scale simulations of Cosmic Dawn and Reionization
• The first power spectrum limit on the 21-cm signal of neutral hydrogen during the Cosmic Dawn at z=20-25 from LOFAR
• First Season MWA Phase II EoR Power Spectrum Results at Redshift 7
• Distributed Radio Interferometric Calibration
• Characterizing Signal Loss in the 21 cm Reionization Power Spectrum: A Revised Study of PAPER-64
• First upper limits on the 21 cm signal power spectrum from cosmic dawn from one night of observations with NenuFAR
• A compression scheme for radio data in high performance computing
• Compression of interferometric radio-astronomical data
• Quantifying Suppression of the Cosmological 21-cm Signal due to Direction Dependent Gain Calibration in Radio Interferometers
• LOFAR detections of low-frequency radio recombination lines towards Cassiopeia A
• Transient RFI environment of LOFAR-LBA at 72-75 MHz: Impact on ultra-widefield AARTFAAC Cosmic Explorer observations of the redshifted 21-cm signal
• Ionospheric contributions to the excess power in high-redshift 21-cm power-spectrum observations with LOFAR

Cited by in corpus (3)

• Improved upper limits on the 21-cm signal power spectrum at $z=17.0$ and $z=20.3$ from an optimal field observed with NenuFAR
• First upper limits on the 21-cm signal power spectrum of neutral hydrogen at $z=9.16$ from the LOFAR 3C196 field
• Lossless compression of simulated radio interferometric visibilities