100 citations · 197 across the 23 of their papers we have counts for
Showing 2022 · cs.CLShow all
2 papers · 2 filters
cs.CL2022★ 2 cited
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky +1
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative position…
cs.CL2022★ 9 cited
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge +1
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propos…