1k citations · 2.5k across the 44 of their papers we have counts for
8 papers · 1 filter
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Yi Tay, Mostafa Dehghani, Samira Abnar +7
There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of…
Retrieval-Enhanced Machine Learning
Hamed Zamani, Fernando Diaz, Mostafa Dehghani +2
Although information access systems have long supported people in accomplishing a wide range of tasks, we propose broadening the scope of users of information access systems to inc…
The Benchmark Lottery
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko +5
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes…
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
Dara Bahri, Heinrich Jiang, Yi Tay +1
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders o…
Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
Dara Bahri, Heinrich Jiang, Yi Tay +1
Detecting out-of-distribution (OOD) examples is critical in many applications. We propose an unsupervised method to detect OOD samples using a -NN density estimate with respect…
Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay, Mostafa Dehghani, Samira Abnar +7
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transf…
Efficient Transformers: A Survey
Yi Tay, Mostafa Dehghani, Dara Bahri +1
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the…
Sparse Sinkhorn Attention
Yi Tay, Dara Bahri, Liu Yang +2
We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend. Our method is based on differentiable sorting of internal representations. Concretel…