A compression scheme for radio data in high performance computing
arXiv:1503.00638 · doi:10.1016/j.ascom.2015.07.002
Abstract
We present a procedure for efficiently compressing astronomical radio data for high performance applications. Integrated, post-correlation data are first passed through a nearly lossless rounding step which compares the precision of the data to a generalized and calibration-independent form of the radiometer equation. This allows the precision of the data to be reduced in a way that has an insignificant impact on the data. The newly developed Bitshuffle lossless compression algorithm is subsequently applied. When the algorithm is used in conjunction with the HDF5 library and data format, data produced by the CHIME Pathfinder telescope is compressed to 28% of its original size and decompression throughputs in excess of 1 GB/s are obtained on a single core.
arXiv comment12 pages, 3 figures, 2 tables. As published in Astronomy and Computing special issue on "The future of astronomical data formats"
References in corpus (9)
- LOFAR: The LOw-Frequency ARray
- The Precision Array for Probing the Epoch of Reionization: 8 Station Results
- Science with ASKAP - the Australian Square Kilometre Array Pathfinder
- The Murchison Widefield Array: Design Overview
- What Next-Generation 21 cm Power Spectrum Measurements Can Teach Us About the Epoch of Reionization
- HI intensity mapping : a single dish approach
- The Baryon Acoustic Oscillation Broadband and Broad-beam Array: Design Overview and Sensitivity Forecasts
- The Tianlai project: a 21cm cosmology experiment
- MeerKAT Key Project Science, Specifications, and Proposals
Cited by in corpus (12)
- Detection of Cosmological 21 cm Emission with the Canadian Hydrogen Intensity Mapping Experiment
- Compression of interferometric radio-astronomical data
- DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device
- Hydrogen Epoch of Reionization Array (HERA) Phase II Deployment and Commissioning
- Strategies for on-chip digital data compression for X-ray pixel detectors
- A clock stabilization system for CHIME/FRB Outriggers
- HDFITS: porting the FITS data model to HDF5
- A lightweight, user-configurable detector ASIC digital architecture with on-chip data compression for MHz X-ray coherent diffraction imaging
- The impact of lossy data compression on the power spectrum of the high redshift 21-cm signal with LOFAR
- Holographic Beam Measurements of the Canadian Hydrogen Intensity Mapping Experiment (CHIME)
- An Overview of CHIME, the Canadian Hydrogen Intensity Mapping Experiment
- Prospects for Direct Electron Detectors in Ultrafast Electron Diffraction and Scattering Experiments