158 citations · 637 across the 25 of their papers we have counts for
25 papers
Bridging Geometric States via Geometric Diffusion Bridge
Shengjie Luo, Yixian Xu, Di He +3
The accurate prediction of geometric state evolution in complex systems is critical for advancing scientific domains such as quantum chemistry and material modeling. Traditional ex…
GeoMFormer: A General Architecture for Geometric Molecular Representation Learning
Tianlang Chen, Shengjie Luo, Di He +3
Molecular modeling, a central topic in quantum mechanics, aims to accurately calculate the properties and simulate the behaviors of molecular systems. The molecular model is govern…
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang +89
In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences. This could herald a new era of scient…
Overcoming the Barrier of Orbital-Free Density Functional Theory for Molecular Systems Using Deep Learning
He Zhang, Siyuan Liu, Jiacheng You +6
Orbital-free density functional theory (OFDFT) is a quantum chemistry formulation that has a lower cost scaling than the prevailing Kohn-Sham DFT, which is increasingly desired for…
Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learning
Shuxin Zheng, Jiyan He, Chang Liu +15
Advances in deep learning have greatly improved structure prediction of molecules. However, many macroscopic observations that are important for real-world applications are not fun…
Invertible Rescaling Network and Its Extensions
Mingqing Xiao, Shuxin Zheng, Chang Liu +2
Image rescaling is a commonly used bidirectional operation, which first downscales high-resolution images to fit various display screens or to be storage- and bandwidth-friendly, a…
One Transformer Can Understand Both 2D & 3D Molecular Data
Shengjie Luo, Tianlang Chen, Yixian Xu +4
Unlike vision and language data which usually has a unique format, molecules can naturally be characterized using different chemical formulations. One can view a molecule as a 2D g…
From Static to Dynamic Structures: Improving Binding Affinity Prediction with Graph-Based Deep Learning
Yaosen Min, Ye Wei, Peizhuo Wang +10
Accurate prediction of protein-ligand binding affinities is an essential challenge in structure-based drug design. Despite recent advances in data-driven methods for affinity predi…
Quantized Training of Gradient Boosting Decision Trees
Yu Shi, Guolin Ke, Zhuoming Chen +2
Recent years have witnessed significant success in Gradient Boosting Decision Trees (GBDT) for a wide range of machine learning applications. Generally, a consensus about GBDT's tr…
Your Transformer May Not be as Powerful as You Expect
Shengjie Luo, Shanda Li, Shuxin Zheng +3
Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far…
An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets
Yu Shi, Shuxin Zheng, Guolin Ke +7
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. The "Graphormer-V…
Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets
Yu Shi, Shuxin Zheng, Guolin Ke +7
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. With these simple…
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
Chengxuan Ying, Mingqi Yang, Shuxin Zheng +7
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track. We adopt Graphormer and ExpC as our basic models. We train each mode…
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Shengjie Luo, Shanda Li, Tianle Cai +6
The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity. Many works focus on approximating the…
Do Transformers Really Perform Bad for Graph Representation?
Chengxuan Ying, Tianle Cai, Shengjie Luo +5
The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance…
How could Neural Networks understand Programs?
Dinglan Peng, Shuxin Zheng, Yatao Li +3
Semantic understanding of programs is a fundamental problem for programming language processing (PLP). Recent works that learn representations of code based on pre-training techniq…
Revisiting Language Encoding in Learning Multilingual Representations
Shengjie Luo, Kaiyuan Gao, Shuxin Zheng +4
Transformer has demonstrated its great power to learn contextual word representations for multiple languages in a single model. To process multilingual sentences in the model, a le…
Modeling Lost Information in Lossy Image Compression
Yaolong Wang, Mingqing Xiao, Chang Liu +2
Lossy image compression is one of the most commonly used operators for digital images. Most recently proposed deep-learning-based image compression methods leverage the auto-encode…
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Zhenhui Xu, Linyuan Gong, Guolin Ke +5
Pre-trained contextual representations (e.g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks. However, large-scale pre-training is computati…
Invertible Image Rescaling
Mingqing Xiao, Shuxin Zheng, Chang Liu +6
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover t…
Cross-Iteration Batch Normalization
Zhuliang Yao, Yue Cao, Shuxin Zheng +2
A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics…
On Layer Normalization in the Transformer Architecture
Ruibin Xiong, Yunchang Yang, Di He +7
The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is sh…
Capacity Control of ReLU Neural Networks by Basis-path Norm
Shuxin Zheng, Qi Meng, Huishuai Zhang +3
Recently, path norm was proposed as a new capacity measure for neural networks with Rectified Linear Unit (ReLU) activation function, which takes the rescaling-invariant property o…
-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Qi Meng, Shuxin Zheng, Huishuai Zhang +3
It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant. Conventional algorithms like stochastic gradient desce…