1k citations · 2.5k across the 44 of their papers we have counts for
16 papers · 1 filter
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification
Jai Gupta, Yi Tay, Chaitanya Kamath +5
With the devastating outbreak of COVID-19, vaccines are one of the crucial lines of defense against mass infection in this global pandemic. Given the protection they provide, vacci…
DSI++: Updating Transformer Memory with New Documents
Sanket Vaibhav Mehta, Jai Gupta, Yi Tay +6
Differentiable Search Indices (DSIs) encode a corpus of documents in model parameters and use the same model to answer user queries directly. Despite the strong performance of DSI…
Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
Bernd Bohnet, Vinh Q. Tran, Pat Verga +19
Large language models (LLMs) have shown impressive results while requiring little or no direct supervision. Further, there is mounting evidence that LLMs may have potential in info…
Transcending Scaling Laws with 0.1% Extra Compute
Yi Tay, Jason Wei, Hyung Won Chung +13
Scaling language models improves performance but comes with significant computational costs. This paper proposes UL2R, a method that substantially improves existing language models…
Retrieval Augmentation for T5 Re-ranker using External Sources
Kai Hui, Tao Chen, Zhen Qin +4
Retrieval augmentation has shown promising improvements in different tasks. However, whether such augmentation can assist a large language model based re-ranker remains unclear. We…
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Yi Tay, Mostafa Dehghani, Samira Abnar +7
There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of…
Confident Adaptive Language Modeling
Tal Schuster, Adam Fisch, Jai Gupta +5
Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in th…
Emergent Abilities of Large Language Models
Jason Wei, Yi Tay, Rishi Bommasani +13
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictabl…
Retrieval-Enhanced Machine Learning
Hamed Zamani, Fernando Diaz, Mostafa Dehghani +2
Although information access systems have long supported people in accomplishing a wide range of tasks, we propose broadening the scope of users of information access systems to inc…
UL2: Unifying Language Learning Paradigms
Yi Tay, Mostafa Dehghani, Vinh Q. Tran +11
Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-traini…
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
Kai Hui, Honglei Zhuang, Tao Chen +8
State-of-the-art neural models typically encode document-query pairs using cross-attention for re-ranking. To this end, models generally utilize an encoder-only (like BERT) paradig…
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters
Tal Schuster, Sihao Chen, Senaka Buthpitiya +2
Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. While early work iden…
HyperPrompt: Prompt-based Task-Conditioning of Transformers
Yun He, Huaixiu Steven Zheng, Yi Tay +9
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way. Here, we explore the use of HyperNetworks to generate hyper-prompts: we pro…
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers
Alyssa Lees, Vinh Q. Tran, Yi Tay +4
On the world wide web, toxic content detectors are a crucial line of defense against potentially hateful and offensive messages. As such, building highly effective classifiers that…
Transformer Memory as a Differentiable Search Index
Yi Tay, Vinh Q. Tran, Mostafa Dehghani +10
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of th…
Atomized Search Length: Beyond User Models
John Alex, Keith Hall, Donald Metzler
We argue that current IR metrics, modeled on optimizing user experience, measure too narrow a portion of the IR space. If IR systems are weak, these metrics undersample or complete…