25 citations · 63 across the 6 of their papers we have counts for
6 papers
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov +7
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These inc…
Is Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?
Chuwei Wang, Shanda Li, Di He +1
The Physics-Informed Neural Network (PINN) approach is a new and promising way to solve partial differential equations using deep learning. The Physics-Informed Loss is the d…
Your Transformer May Not be as Powerful as You Expect
Shengjie Luo, Shanda Li, Shuxin Zheng +3
Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far…
Learning Physics-Informed Neural Networks without Stacked Back-propagation
Di He, Shanda Li, Wenlei Shi +5
Physics-Informed Neural Network (PINN) has become a commonly used machine learning approach to solve partial differential equations (PDE). But, facing high-dimensional secondorder…
Can Vision Transformers Perform Convolution?
Shanda Li, Xiangning Chen, Di He +1
Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vi…
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Shengjie Luo, Shanda Li, Tianle Cai +6
The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity. Many works focus on approximating the…