ClimaX: A foundation model for weather and climate

Abstract

Most state-of-the-art approaches for weather and climate modeling are based on physics-informed numerical models of the atmosphere. These approaches aim to model the non-linear dynamics and complex interactions between multiple variables, which are challenging to approximate. Additionally, many such numerical models are computationally intensive, especially when modeling the atmospheric phenomenon at a fine-grained spatial and temporal resolution. Recent data-driven approaches based on machine learning instead aim to directly solve a downstream forecasting or projection task by learning a data-driven functional mapping using deep neural networks. However, these networks are trained using curated and homogeneous climate datasets for specific spatiotemporal tasks, and thus lack the generality of numerical models. We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatio-temporal coverage, and physical groundings. ClimaX extends the Transformer architecture with novel encoding and aggregation blocks that allow effective use of available compute while maintaining general utility. ClimaX is pre-trained with a self-supervised learning objective on climate datasets derived from CMIP6. The pre-trained ClimaX can then be fine-tuned to address a breadth of climate and weather tasks, including those that involve atmospheric variables and spatio-temporal scales unseen during pretraining. Compared to existing data-driven baselines, we show that this generality in ClimaX results in superior performance on benchmarks for weather forecasting and climate projections, even when pretrained at lower resolutions and compute budgets. The source code is available at https://github.com/microsoft/ClimaX.

International Conference on Machine Learning 2023

Cited by in corpus (12)

• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• AstroCLIP: A Cross-Modal Foundation Model for Galaxies
• A Machine Learning Tutorial for Operational Meteorology, Part II: Neural Networks and Deep Learning
• Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events
• A Practical Probabilistic Benchmark for AI Weather Models
• Transferring climate change physical knowledge
• Lightning-Fast Convective Outlooks: Predicting Severe Convective Environments with Global AI-based Weather Models
• Stochastic Latent Transformer: Efficient Modelling of Stochastically Forced Zonal Jets
• A Spatiotemporal Radar-Based Precipitation Model for Water Level Prediction and Flood Forecasting
• Seasonal forecasting using the GenCast probabilistic machine learning model
• Implicit Neural Representations for Simultaneous Reduction and Continuous Reconstruction of Multi-Altitude Climate Data
• Finding the Perfect Fit: Applying Regression Models to ClimateBench v1.0