3 citations · 9 across the 9 of their papers we have counts for
9 papers
Emerging Flexible Designs for Geospatial Multimodal Foundation Models
Philipe Dias, Waqwoya Abebe, Abhishek Potnis +4
Foundation models are rapidly transforming Earth observation by enabling scalable pretraining across diverse unlabeled geospatial modalities. However, their architectural diversity…
Towards Scaling Law Analysis For Spatiotemporal Weather Data
Alexander Kiefer, Prasanna Balaprakash, Xiao Wang
Compute-optimal scaling laws are relatively well studied for NLP and CV, where objectives are typically single-step and targets are comparatively homogeneous. Weather forecasting i…
Global Attention with Linear Complexity for Exascale Generative Data Assimilation in Earth System Prediction
Xiao Wang, Zezhong Zhang, Isaac Lyngaas +10
Accurate Earth system prediction requires state inference from incomplete observations, but conventional two-stage data assimilation (DA) is computationally prohibitive because rep…
Distributed Cross-Channel Hierarchical Aggregation for Foundation Models
Aristeidis Tsaris, Isaac Lyngaas, John Lagregren +6
Vision-based scientific foundation models hold significant promise for advancing scientific discovery and innovation. This potential stems from their ability to aggregate images fr…
ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling
Xiao Wang, Jong-Youl Choi, Takuya Kurihaya +15
Sparse observations and coarse-resolution climate models limit effective regional decision-making, underscoring the need for robust downscaling. However, existing AI methods strugg…
Sequence Length Scaling in Vision Transformers for Scientific Images on Frontier
Aristeidis Tsaris, Chengming Zhang, Xiao Wang +9
Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence length…
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability
Xiao Wang, Siyan Liu, Aristeidis Tsaris +8
Earth system predictability is challenged by the complexity of environmental dynamics and the multitude of variables involved. Current AI foundation models, although advanced by le…
Optimizing Distributed Training on Frontier for Large Language Models
Sajal Dash, Isaac Lyngaas, Junqi Yin +5
Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Recent studies on loss sca…
Ultra-Long Sequence Distributed Transformer
Xiao Wang, Isaac Lyngaas, Aristeidis Tsaris +7
Transformer models trained on long sequences often achieve higher accuracy than short sequences. Unfortunately, conventional transformers struggle with long sequence training due t…