STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Abstract

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality -- even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7$\times$ faster than SZ3.

accepted by SC '25

References in corpus (9)

• Formation of Hard Power-laws in the Energetic Particle Spectra Resulting from Relativistic Magnetic Reconnection
• Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization
• TTHRESH: Tensor Compression for Multidimensional Visual Data
• cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data
• MGARD: A multigrid framework for high-performance, error-controlled data compression and refactoring
• AMM: Adaptive Multilinear Meshes
• TAC: Optimizing Error-Bounded Lossy Compression for Three-Dimensional Adaptive Mesh Refinement Simulations
• On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective
• TAC+: Optimizing Error-Bounded Lossy Compression for 3D AMR Simulations