Lossless compression of simulated radio interferometric visibilities

Abstract

Context. Processing radio interferometric data often requires storing forward-predicted model data. In direction-dependent calibration, these data may have a volume an order of magnitude larger than the original data. Existing lossy compression techniques work well for observed, noisy data, but cause issues in calibration when applied to forward-predicted model data. Aims. To reduce the volume of forward-predicted model data, we present a lossless compression method called Simulated Signal Compression (Sisco) for noiseless data that integrates seamlessly with existing workflows. We show that Sisco can be combined with baseline-dependent averaging for further size reduction. Methods. Sisco decomposes complex floating-point visibility values and uses polynomial extrapolation in time and frequency to predict values, groups bytes for efficient encoding, and compresses residuals using the Deflate algorithm. We evaluate Sisco on diverse LOFAR, MeerKAT, and MWA datasets with various extrapolation functions. Implemented as an open-source Casacore storage manager, it can directly be used by any observatory that makes use of this format. Results. We find that a combination of linear and quadratic prediction yields optimal compression, reducing noiseless forward-predicted model data to 24% of its original volume on average. Compression varies by dataset, ranging from 13% for smooth data to 38% for less predictable data. For pure noise data, compression achieves just a size of 84% due to the unpredictability of such data. With the current implementation, the achieved compression throughput is with 534 MB/s mostly dominated by I/O on our testing platform, but occupies the processor during compression or decompression. Finally, we discuss the extension to a lossy algorithm.

Accepted for publication in A&A. 8 pages, 3 figures

References in corpus (24)

• LOFAR: The LOw-Frequency ARray
• WSClean: an implementation of a fast, generic wide-field imager for radio astronomy
• Systematic effects in LOFAR data: A unified calibration strategy
• Real-Time Calibration of the Murchison Widefield Array
• LOFAR Detection of 110-188 MHz Emission and Frequency-Dependent Activity from FRB 20180916B
• LOFAR observations of galaxy clusters in HETDEX
• Applying full polarization A-Projection to very wide field of view instruments: An imager for LOFAR
• Distributed Radio Interferometric Calibration
• Sub-arcsecond imaging with the International LOFAR Telescope I. Foundational calibration strategy and pipeline
• Image Domain Gridding: a fast method for convolutional resampling of visibilities
• The Faraday Rotation Measure Grid of the LOFAR Two-metre Sky Survey: Data Release 2
• Lossless Astronomical Image Compression and the Effects of Noise
• Compression of interferometric radio-astronomical data
• Carbon and hydrogen radio recombination lines from the cold clouds towards Cassiopeia A
• MGARD: A multigrid framework for high-performance, error-controlled data compression and refactoring
• Baseline Dependent Averaging in Radio Interferometry
• Assessing the impact of two independent direction-dependent calibration algorithms on the LOFAR 21-cm signal power spectrum
• Optimal Compression of Floating-point Astronomical Images Without Significant Loss of Information
• Simulating the 21cm forest detectable with LOFAR and SKA in the spectra of high-z GRBs
• MIGHTEE-HI: deep spectral line observations of the COSMOS field
• The prototypical major cluster merger Abell 754. I. Calibration of MeerKAT data and radio/X-ray spectral mapping of the cluster
• An interference detection strategy for Apertif based on AOFlagger 3
• The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR
• Optimising the Processing and Storage of Visibilities using lossy compression