Enabling near real-time remote search for fast transient events with lossy data compression

Abstract

We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster program (DWF). DWF uses ~20 telescopes from radio to gamma-rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. DWF search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of DWF led by the Dark Energy Camera (DECam) at CTIO. Each DECam image has 70 total CCDs saved as a ~1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications -- in minutes for rapid follow-up triggers on other telescopes -- requires computational power exceeding what is currently available on-site at CTIO. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualization and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression -- keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ~25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customized compression and decompression steps to the science pipeline considerably reduces the transmission time -- validating its introduction to the DWF science pipeline and enabling science that was otherwise too difficult with current technology.

arXiv comment13 pages, 9 figures, 2 tables. Accepted for publication by the Publications of the Astronomical Society of Australia

References in corpus (9)

• A bright millisecond radio burst of extragalactic origin
• Transient Events from Neutron Star Mergers
• The Dark Energy Camera
• Testing LMC Microlensing Scenarios: The Discrimination Power of the SuperMACHO Microlensing Survey
• A Comparison of Intermediate Mass Black Hole Candidate ULXs and Stellar-Mass Black Holes
• Data Compression for the Tomo-e Gozen with Low-rank Matrix Approximation
• Optimal Compression of Floating-point Astronomical Images Without Significant Loss of Information
• Mary, a pipeline to aid discovery of optical transients
• Collaborative workspaces to accelerate discovery

Cited by in corpus (5)

• Follow up of GW170817 and its electromagnetic counterpart by Australian-led observing programs
• Unsupervised machine learning for transient discovery in Deeper, Wider, Faster light curves
• The Deeper Wider Faster program: chasing the fastest bursts in the Universe
• Mary, a pipeline to aid discovery of optical transients
• Finding Fast Transients in Real Time Using Novel Light Curve Analysis Algorithm