132 citations · 284 across the 21 of their papers we have counts for
22 papers
Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
Yixian Xu, Yuanrui Zhang, Shengjie Luo +2
Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards. However, current RL algorithms for diffu…
In-Place Test-Time Training
Guhao Feng, Shengjie Luo, Kai Hua +4
The static ``train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continuous streams of new informat…
Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value
Yixian Xu, Shengjie Luo, Liwei Wang +2
Diffusion models have achieved remarkable success in generative modeling. Despite more stable training, the loss of diffusion models is not indicative of absolute data-fitting qual…
UniSite: The First Cross-Structure Dataset and Learning Framework for End-to-End Ligand Binding Site Detection
Jigang Fan, Quanlin Wu, Shengjie Luo +1
The detection of ligand binding sites for proteins is a fundamental step in Structure-Based Drug Design. Despite notable advances in recent years, existing methods, datasets, and e…
Bridging Geometric States via Geometric Diffusion Bridge
Shengjie Luo, Yixian Xu, Di He +3
The accurate prediction of geometric state evolution in complex systems is critical for advancing scientific domains such as quantum chemistry and material modeling. Traditional ex…
Physical Consistency Bridges Heterogeneous Data in Molecular Multi-Task Learning
Yuxuan Ren, Dihan Zheng, Chang Liu +7
In recent years, machine learning has demonstrated impressive capability in handling molecular science tasks. To support various molecular properties at scale, machine learning mod…
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
Guhao Feng, Kai Yang, Yuntian Gu +6
Despite the remarkable success of Transformer-based large language models (LLMs) across various domains, understanding and enhancing their mathematical capabilities remains a signi…
Let the Code LLM Edit Itself When You Edit the Code
Zhenyu He, Jun Zhang, Shengjie Luo +3
In this work, we investigate a typical scenario in code generation where a developer edits existing code in real time and requests a code assistant, e.g., a large language model, t…
GeoMFormer: A General Architecture for Geometric Molecular Representation Learning
Tianlang Chen, Shengjie Luo, Di He +3
Molecular modeling, a central topic in quantum mechanics, aims to accurately calculate the properties and simulate the behaviors of molecular systems. The molecular model is govern…
Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
Zhenyu He, Guhao Feng, Shengjie Luo +6
In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position,…
Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products
Shengjie Luo, Tianlang Chen, Aditi S. Krishnapriyan
Developing equivariant neural networks for the E(3) group plays an important role in modeling 3D data across real-world applications. Enforcing this equivariance primarily involves…
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov +7
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These inc…
Rethinking the Expressive Power of GNNs via Graph Biconnectivity
Bohang Zhang, Shengjie Luo, Liwei Wang +1
Designing expressive Graph Neural Networks (GNNs) is a central topic in learning graph-structured data. While numerous approaches have been proposed to improve GNNs in terms of the…
One Transformer Can Understand Both 2D & 3D Molecular Data
Shengjie Luo, Tianlang Chen, Yixian Xu +4
Unlike vision and language data which usually has a unique format, molecules can naturally be characterized using different chemical formulations. One can view a molecule as a 2D g…
Your Transformer May Not be as Powerful as You Expect
Shengjie Luo, Shanda Li, Shuxin Zheng +3
Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far…
An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets
Yu Shi, Shuxin Zheng, Guolin Ke +7
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. The "Graphormer-V…
Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets
Yu Shi, Shuxin Zheng, Guolin Ke +7
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. With these simple…
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
Chengxuan Ying, Mingqi Yang, Shuxin Zheng +7
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track. We adopt Graphormer and ExpC as our basic models. We train each mode…
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Shengjie Luo, Shanda Li, Tianle Cai +6
The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity. Many works focus on approximating the…
Do Transformers Really Perform Bad for Graph Representation?
Chengxuan Ying, Tianle Cai, Shengjie Luo +5
The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance…
Revisiting Language Encoding in Learning Multilingual Representations
Shengjie Luo, Kaiyuan Gao, Shuxin Zheng +4
Transformer has demonstrated its great power to learn contextual word representations for multiple languages in a single model. To process multilingual sentences in the model, a le…
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
Tianle Cai, Shengjie Luo, Keyulu Xu +3
Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what…