Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data
arXiv:2307.04216 · doi:10.69709/CAIC.2024.193132
Abstract
Lossy compression has become an important technique to reduce data size in many domains. This type of compression is especially valuable for large-scale scientific data, whose size ranges up to several petabytes. Although Autoencoder-based models have been successfully leveraged to compress images and videos, such neural networks have not widely gained attention in the scientific data domain. Our work presents a neural network that not only significantly compresses large-scale scientific data, but also maintains high reconstruction quality. The proposed model is tested with scientific benchmark data available publicly and applied to a large-scale high-resolution climate modeling data set. Our model achieves a compression ratio of 140 on several benchmark data sets without compromising the reconstruction quality. 2D simulation data from the High-Resolution Community Earth System Model (CESM) Version 1.3 over 500 years are also being compressed with a compression ratio of 200 while the reconstruction error is negligible for scientific analysis.
14 pages
References in corpus (8)
- Auto-Encoding Variational Bayes
- Gaussian Error Linear Units (GELUs)
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Variational image compression with a scale hyperprior
- End-to-end Optimized Image Compression
- NVAE: A Deep Hierarchical Variational Autoencoder
- Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization
- QMCPACK : An open source ab initio Quantum Monte Carlo package for the electronic structure of atoms, molecules, and solids