Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding

Abstract

Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e.g., pixel positions on an image, where $L_2$ distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.

arXiv comment35th Conference on Neural Information Processing Systems (NeurIPS 2021)

References in corpus (11)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Generating Long Sequences with Sparse Transformers
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Reformer: The Efficient Transformer
• Encoding word order in complex embeddings
• Compact Random Feature Maps
• Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels
• Recurrent Space-time Graph Neural Networks
• But How Does It Work in Theory? Linear SVM with Random Features
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
• Fastfood: Approximate Kernel Expansions in Loglinear Time

Cited by in corpus (1)

• Gaussian Radar Transformer for Semantic Segmentation in Noisy Radar Data