Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization
arXiv:1706.03791 · doi:10.1109/IPDPS.2017.115
Abstract
Today's HPC applications are producing extremely large amounts of data, such that data storage and analysis are becoming more challenging for scientific research. In this work, we design a new error-controlled lossy compression algorithm for large-scale scientific data. Our key contribution is significantly improving the prediction hitting rate (or prediction accuracy) for each data point based on its nearby data values along multiple dimensions. We derive a series of multilayer prediction formulas and their unified formula in the context of data compression. One serious challenge is that the data prediction has to be performed based on the preceding decompressed values during the compression in order to guarantee the error bounds, which may degrade the prediction accuracy in turn. We explore the best layer for the prediction by considering the impact of compression errors on the prediction accuracy. Moreover, we propose an adaptive error-controlled quantization encoder, which can further improve the prediction hitting rate considerably. The data size can be reduced significantly after performing the variable-length encoding because of the uneven distribution produced by our quantization encoder. We evaluate the new compressor on production scientific data sets and compare it with many other state-of-the-art compressors: GZIP, FPZIP, ZFP, SZ-1.1, and ISABELA. Experiments show that our compressor is the best in class, especially with regard to compression factors (or bit-rates) and compression errors (including RMSE, NRMSE, and PSNR). Our solution is better than the second-best solution by more than a 2x increase in the compression factor and 3.8x reduction in the normalized root mean squared error on average, with reasonable error bounds and user-desired bit-rates.
Accepted by IPDPS'17, 11 pages, 10 figures, double column
Cited by in corpus (34)
- Full-State Quantum Circuit Simulation by Using Data Compression
- SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors
- cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data
- DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression
- Z-checker: A Framework for Assessing Lossy Compression of Scientific Data
- Understanding GPU-Based Lossy Compression for Extreme-Scale Cosmological Simulations
- Improving Performance of Iterative Methods by Lossy Checkponting
- FZ-GPU: A Fast and High-Ratio Lossy Compressor for Scientific Computing Applications on GPUs
- AMM: Adaptive Multilinear Meshes
- GPULZ: Optimizing LZSS Lossless Compression for Multi-byte Data on Modern GPUs
- Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling
- TAC: Optimizing Error-Bounded Lossy Compression for Three-Dimensional Adaptive Mesh Refinement Simulations
- On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective
- TAC+: Optimizing Error-Bounded Lossy Compression for 3D AMR Simulations
- Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis
- Lossy compression of statistical data using quantum annealer
- CEAZ: Accelerating Parallel I/O via Hardware-Algorithm Co-Designed Adaptive Lossy Compression
- A General Framework for Augmenting Lossy Compressors with Topological Guarantees
- Compression challenges in large scale PDE solvers
- WaveRange: Wavelet-based data compression for three-dimensional numerical simulations on regular grids
- MGARD+: Optimizing Multilevel Methods for Error-bounded Scientific Data Reduction
- SDC Resilient Error-bounded Lossy Compressor
- Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data
- COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
- SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors
- Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP
- STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data
- Exploring Autoencoder-based Error-bounded Compression for Scientific Data
- Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures
- Pass-efficient methods for compression of high-dimensional turbulent flow data
- LCFI: A Fault Injection Tool for Studying Lossy Compression Error Propagation in HPC Programs
- Deterministic matrix sketches for low-rank compression of high-dimensional simulation data
- Accelerating In-transit Isosurface Generation With Topology Preserving Compression
- A framework for compressing unstructured scientific data via serialization