Relative Position Prediction as Pre-training for Text Encoders

Rickard Brüel-Gabrielsson, Chris Scarvelis

Meaning is defined by the company it keeps. However, company is two-fold: It's based on the identity of tokens and also on their position (topology). We argue that a position-centr…