Rethinking Positional Encoding

Abstract

It is well noted that coordinate based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been solely studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice. Codes available at https://github.com/osiriszjq/Rethinking-positional-encoding.

References in corpus (11)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• On the Spectral Bias of Neural Networks
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
• Rethinking Positional Encoding in Language Pre-training
• Constituency Parsing with a Self-Attentive Encoder
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

Cited by in corpus (6)

• Coordinate-Based Seismic Interpolation in Irregular Land Survey: A Deep Internal Learning Approach
• Polynomial Neural Fields for Subband Decomposition and Manipulation
• Neural Fields in Visual Computing and Beyond
• IntraSeismic: a coordinate-based learning approach to seismic inversion
• An Empirical Study: Extensive Deep Temporal Point Process
• Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fields