100 citations · 197 across the 23 of their papers we have counts for
3 papers · 1 filter
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky +1
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative position…
Training Discrete Deep Generative Models via Gapped Straight-Through Estimator
Ting-Han Fan, Ta-Chung Chi, Alexander I. Rudnicky +1
While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains c…
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge +1
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propos…