19 citations · 63 across the 20 of their papers we have counts for
4 papers · 1 filter
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky +1
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative position…
On Task-Adaptive Pretraining for Dialogue Response Selection
Tzu-Hsiang Lin, Ta-Chung Chi, Anna Rumshisky
Recent advancements in dialogue response selection (DRS) are based on the \textit{task-adaptive pre-training (TAP)} approach, by first initializing their model with BERT~\cite{devl…
Training Discrete Deep Generative Models via Gapped Straight-Through Estimator
Ting-Han Fan, Ta-Chung Chi, Alexander I. Rudnicky +1
While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains c…
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge +1
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propos…