Data Compression in the Petascale Astronomy Era: a GERLUMPH case study
arXiv:1505.05617 · doi:10.1016/j.ascom.2015.05.003
Abstract
As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -- and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. However, there is still a need to evaluate its potential on other type of astronomical data, such as from numerical simulations. GERLUMPH (the GPU-Enabled High Resolution cosmological MicroLensing parameter survey) represents an example of a data intensive project in theoretical astrophysics. In the next phase of processing, the ~27 terabyte GERLUMPH dataset is set to grow by a factor of 100 -- well beyond the current storage capabilities of the supercomputing facility on which it resides. In order to minimise bandwidth usage, file transfer time, and storage space, this work evaluates several data compression techniques. Specifically, we investigate off-the-shelf and custom lossless compression algorithms as well as the lossy JPEG2000 compression format. Results of lossless compression algorithms on GERLUMPH data products show small compression ratios (1.35:1 to 4.69:1 of input file size) varying with the nature of the input data. Our results suggest that JPEG2000 could be suitable for other numerical datasets stored as gridded data or volumetric data. When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. In our case study, lossy compression and a high compression ratio do not significantly compromise the intended use of the data for constraining quasar source profiles from cosmological microlensing.
arXiv comment15 pages, 9 figures, 5 tables. Published in the Special Issue of Astronomy & Computing on The future of astronomical data formats
References in corpus (21)
- Simulating the joint evolution of quasars, galaxies and their large-scale distribution
- The Ninth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-III Baryon Oscillation Spectroscopic Survey
- Gravitationally lensed quasars and supernovae in future wide-field optical imaging surveys
- Quantitative Interpretation of Quasar Microlensing Light Curves
- Decoding billions of integers per second through vectorization
- The Microlensing Properties of a Sample of 87 Lensed Quasars
- Astrophysical Applications of Gravitational Microlensing
- A microlensing study of the accretion disc in the quasar MG 0414+0534
- The accretion disc in the quasar SDSS J0924+0219
- MeerKAT Key Project Science, Specifications, and Proposals
- Scientific Visualization in Astronomy: Towards the Petascale Astronomy Era
- Teraflop per second gravitational lensing ray-shooting using graphics processing units
- GERLUMPH Data Release 1: High-resolution cosmological microlensing magnification maps and eResearch tools
- Adventures in the microlensing cloud: large datasets, eResearch tools, and GPUs
- Optimal Compression of Floating-point Astronomical Images Without Significant Loss of Information
- Tera-scale Astronomical Data Analysis and Visualization
- A Distributed GPU-based Framework for real-time 3D Volume Rendering of Large Astronomical Data Cubes
- PHOTZIP: A Lossy FITS Image Compression Algorithm that Protects User-Defined Levels of Photometric Integrity
- SkuareView: Client-Server Framework for Accessing Extremely Large Radio Astronomy Image Data
- What bandwidth do I need for my image?
- Application of Lossless Data Compression Techniques to Radio Astronomy Data flows