CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

Abstract

Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters.

References in corpus (18)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Neural Machine Translation by Jointly Learning to Align and Translate
• mixup: Beyond Empirical Risk Minimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Language Models are Few-Shot Learners
• Is Space-Time Attention All You Need for Video Understanding?
• Conditional Positional Encodings for Vision Transformers
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• CvT: Introducing Convolutions to Vision Transformers
• Transformers with convolutional context for ASR
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• Very Deep Transformers for Neural Machine Translation
• Rethinking Positional Encoding in Language Pre-training
• Encoding word order in complex embeddings
• MultiGrain: a unified image embedding for classes and instances

Cited by in corpus (2)

• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP