Compression of interferometric radio-astronomical data

Abstract

The volume of radio-astronomical data is a considerable burden in the processing and storing of radio observations with high time and frequency resolutions and large bandwidths. Lossy compression of interferometric radio-astronomical data is considered to reduce the volume of visibility data and to speed up processing. A new compression technique named "Dysco" is introduced that consists of two steps: a normalization step, in which grouped visibilities are normalized to have a similar distribution; and a quantization and encoding step, which rounds values to a given quantization scheme using a dithering scheme. Several non-linear quantization schemes are tested and combined with different methods for normalizing the data. Four data sets with observations from the LOFAR and MWA telescopes are processed with different processing strategies and different combinations of normalization and quantization. The effects of compression are measured in image plane. The noise added by the lossy compression technique acts like normal system noise. The accuracy of Dysco is depending on the signal-to-noise ratio of the data: noisy data can be compressed with a smaller loss of image quality. Data with typical correlator time and frequency resolutions can be compressed by a factor of 6.4 for LOFAR and 5.3 for MWA observations with less than 1% added system noise. An implementation of the compression technique is released that provides a Casacore storage manager and allows transparent encoding and decoding. Encoding and decoding is faster than the read/write speed of typical disks. The technique can be used for LOFAR and MWA to reduce the archival space requirements for storing observed data. Data from SKA-low will likely be compressible by the same amount as LOFAR. The same technique can be used to compress data from other telescopes, but a different bit-rate might be required.

Accepted for publication in A&A. 13 pages, 8 figures. Abstract was abridged

References in corpus (18)

• LOFAR: The LOw-Frequency ARray
• The Murchison Widefield Array: the Square Kilometre Array Precursor at low radio frequencies
• WSClean: an implementation of a fast, generic wide-field imager for radio astronomy
• GaLactic and Extragalactic All-sky Murchison Widefield Array (GLEAM) survey I: A low-frequency extragalactic catalogue
• The Expanded Very Large Array -- a New Telescope for New Science
• The Australia Telescope Compact Array Broadband Backend (CABB)
• GLEAM: The GaLactic and Extragalactic All-sky MWA survey
• LOFAR facet calibration
• Observing pulsars and fast transients with LOFAR
• Revisiting the radio interferometer measurement equation. II. Calibration and direction-dependent effects
• The LOFAR radio environment
• Parametrising Epoch of Reionization foregrounds: A deep survey of low-frequency point-source spectra with the MWA
• A compression scheme for radio data in high performance computing
• Clustered Calibration: An Improvement to Radio Interferometric Direction Dependent Self-Calibration
• High-resolution EVLA image of dimethyl ether (CH${}_3$)${}_2$O in Orion--KL
• Wide-field broadband radio imaging with phased array feeds: a pilot multi-epoch continuum survey with ASKAP-BETA
• Discovery of Carbon Radio Recombination Lines in M82
• Optimal Compression of Floating-point Astronomical Images Without Significant Loss of Information

Cited by in corpus (31)

• The LOFAR Two-metre Sky Survey - II. First data release
• The LOFAR Two-metre Sky Survey -- V. Second data release
• LOFAR observations of galaxy clusters in HETDEX
• The LOFAR LBA Sky Survey I. survey description and preliminary data release
• Sub-arcsecond imaging with the International LOFAR Telescope I. Foundational calibration strategy and pipeline
• The impact of interference excision on 21-cm Epoch of Reionization power spectrum analyses
• First upper limits on the 21 cm signal power spectrum from cosmic dawn from one night of observations with NenuFAR
• First release of Apertif imaging survey data
• Particle re-acceleration and diffuse radio sources in the galaxy cluster Abell 1550
• The prototypical major cluster merger Abell 754. I. Calibration of MeerKAT data and radio/X-ray spectral mapping of the cluster
• High-resolution international LOFAR observations of 4C~43.15 -- Spectral ages and injection indices in a high-z radio galaxy
• Pushing subarcsecond resolution imaging down to 30 MHz with the trans-European International LOFAR Telescope
• LOFAR Detection of a Low-Power Radio Halo in the Galaxy Cluster Abell 990
• The LOFAR Two-metre Sky Survey: Deep Fields Data Release 2. I. The ELAIS-N1 field
• GaLactic and Extragalactic All-sky Murchison Widefield Array (GLEAM) survey III: South Galactic Pole data release
• Improved upper limits on the 21-cm signal power spectrum at $z=17.0$ and $z=20.3$ from an optimal field observed with NenuFAR
• The LOFAR LBA Sky Survey: Deep Fields I. The Boötes Field
• Reaching thermal noise at ultra-low radio frequencies: the Toothbrush radio relic downstream of the shock front
• First upper limits on the 21-cm signal power spectrum of neutral hydrogen at $z=9.16$ from the LOFAR 3C196 field
• The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR
• Strong low-frequency radio flaring from Cygnus X-3 observed with LOFAR
• Learning from the present for the future: the Juelich LOFAR Long-term Archive
• CHEX-MATE: New detections and properties of the radio diffuse emission in massive clusters with MeerKAT
• Optimising the Processing and Storage of Visibilities using lossy compression
• Enhanced radio emission between a galaxy cluster pair
• Fine-grained Distributed Averaging for Large-scale Radio Interferometric Measurement Sets
• LOFAR 58 MHz Legacy Survey of the 3CRR Catalog
• Fast algorithms to approximate the position-dependent point spread function responses in radio interferometric wide-field imaging
• A deep study of A399-401: An application for wide-field facet calibration
• Lossless compression of simulated radio interferometric visibilities
• The resolved jet of 3C 273 at 150 MHz