112 citations · 295 across the 13 of their papers we have counts for
12 papers · 1 filter
Learning from History: A Retrieval-Augmented Framework for Spatiotemporal Prediction
Hao Jia, Penghao Zhao, Hao Wu +3
Accurate and long-term spatiotemporal prediction for complex physical systems remains a fundamental challenge in scientific computing. While deep learning models, as powerful param…
MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
Pinxue Zhao, Hailin Zhang, Fangcheng Fu +9
Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges…
BeamVQ: Aligning Space-Time Forecasting Model via Self-training on Physics-aware Metrics
Hao Wu, Xingjian Shi, Ziyue Huang +6
Data-driven deep learning has emerged as the new paradigm to model complex physical space-time systems. These data-driven methods learn patterns by optimizing statistical metrics a…
Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
Shuaipeng Li, Penghao Zhao, Hailin Zhang +10
In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optim…
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Xiaonan Nie, Yi Liu, Fangcheng Fu +5
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as…
BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
Fangcheng Fu, Huanran Xue, Yong Cheng +2
Due to the rising concerns on privacy protection, how to build machine learning (ML) models over different data sources with security guarantees is gaining more popularity. Vertica…
Graph Attention Multi-Layer Perceptron
Wentao Zhang, Ziqi Yin, Zeang Sheng +6
Graph neural networks (GNNs) have achieved great success in many graph-based applications. However, the enormous size and high sparsity level of graphs hinder their applications un…
PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
Wentao Zhang, Yu Shen, Zheyu Lin +6
Graph neural networks (GNNs) have achieved state-of-the-art performance in various graph-based tasks. However, as mainstream GNNs are designed based on the neural message passing m…
HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
Xupeng Miao, Hailin Zhang, Yining Shi +4
Embedding models have been an effective learning paradigm for high-dimensional data. However, one open issue of embedding models is that their representations (latent factors) ofte…
Node Dependent Local Smoothing for Scalable Graph Learning
Wentao Zhang, Mingyu Yang, Zeang Sheng +5
Recent works reveal that feature or label smoothing lies at the core of Graph Neural Networks (GNNs). Concretely, they show feature smoothing combined with simple linear regression…
Graph Attention MLP with Reliable Label Utilization
Wentao Zhang, Ziqi Yin, Zeang Sheng +5
Graph neural networks (GNNs) have recently achieved state-of-the-art performance in many graph-based applications. Despite the high expressive power, they typically need to perform…
GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing
Wentao Zhang, Yu Shen, Zheyu Lin +6
In recent studies, neural message passing has proved to be an effective way to design graph neural networks (GNNs), which have achieved state-of-the-art performance in many graph-b…