4 papers
Learning from History: A Retrieval-Augmented Framework for Spatiotemporal Prediction
Hao Jia, Penghao Zhao, Hao Wu +3
Accurate and long-term spatiotemporal prediction for complex physical systems remains a fundamental challenge in scientific computing. While deep learning models, as powerful param…
MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
Pinxue Zhao, Hailin Zhang, Fangcheng Fu +9
Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges…
Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
Shuaipeng Li, Penghao Zhao, Hailin Zhang +10
In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optim…
BeamVQ: Aligning Space-Time Forecasting Model via Self-training on Physics-aware Metrics
Hao Wu, Xingjian Shi, Ziyue Huang +6
Data-driven deep learning has emerged as the new paradigm to model complex physical space-time systems. These data-driven methods learn patterns by optimizing statistical metrics a…