1k citations · 2.5k across the 44 of their papers we have counts for
10 papers · 1 filter
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
Yikang Shen, Yi Tay, Che Zheng +3
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the…
Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay, Mostafa Dehghani, Samira Abnar +7
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transf…
Surprise: Result List Truncation via Extreme Value Theory
Dara Bahri, Che Zheng, Yi Tay +2
Work in information retrieval has largely been centered around ranking and relevance: given a query, return some number of results ordered by relevance to the user. The problem of…
Efficient Transformers: A Survey
Yi Tay, Mostafa Dehghani, Dara Bahri +1
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the…
Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study
Dara Bahri, Yi Tay, Che Zheng +3
Large generative language models such as GPT-2 are well-known for their ability to generate text as well as their utility in supervised downstream tasks via fine-tuning. Our work i…
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
Yi Tay, Zhe Zhao, Dara Bahri +2
Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a…
Synthesizer: Rethinking Self-Attention in Transformer Models
Yi Tay, Dara Bahri, Donald Metzler +3
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importan…
Choppy: Cut Transformer For Ranked List Truncation
Dara Bahri, Yi Tay, Che Zheng +2
Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user. However, the proble…
Reverse Engineering Configurations of Neural Text Generation Models
Yi Tay, Dara Bahri, Che Zheng +3
This paper seeks to develop a deeper understanding of the fundamental properties of neural text generations models. The study of artifacts that emerge in machine generated text as…
Sparse Sinkhorn Attention
Yi Tay, Dara Bahri, Liu Yang +2
We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend. Our method is based on differentiable sorting of internal representations. Concretel…