Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

Haris Riaz, Hyungji Kim, Mihai Surdeanu

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed…