221 citations · 587 across the 23 of their papers we have counts for
23 papers
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
Zhichao Xu, Xueguang Ma, Shengyao Zhuang +5
Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure av…
Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality
Xueguang Ma, Luyu Gao, Shengyao Zhuang +3
Recent advancements in large language models (LLMs) have driven interest in billion-scale retrieval models with strong generalization across retrieval tasks and languages. Addition…
Repository-level Code Search with Neural Retrieval Methods
Siddharth Gandhi, Luyu Gao, Jamie Callan
This paper presents a multi-stage reranking system for repository-level code search, which leverages the vastly available commit histories of large open-source repositories to aid…
ACER: Automatic Language Model Context Extension via Retrieval
Luyu Gao, Yunyi Zhang, Jamie Callan
Long-context modeling is one of the critical capabilities of language AI for digesting and reasoning over complex information pieces. In practice, long-context capabilities are typ…
SciCode: A Research Coding Benchmark Curated by Scientists
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang +27
Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evalua…
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions
Vijay Viswanathan, Luyu Gao, Tongshuang Wu +2
Modern machine learning relies on datasets to develop and validate research ideas. Given the growth of publicly available data, finding the right dataset to use is increasingly dif…
Active Retrieval Augmented Generation
Zhengbao Jiang, Frank F. Xu, Luyu Gao +6
Despite the remarkable ability of large language models (LMs) to comprehend and generate language, they have a tendency to hallucinate and create factually inaccurate output. Augme…
Self-Refine: Iterative Refinement with Self-Feedback
Aman Madaan, Niket Tandon, Prakhar Gupta +13
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an…
Precise Zero-Shot Dense Retrieval without Relevance Labels
Luyu Gao, Xueguang Ma, Jimmy Lin +1
While dense retrieval has been shown effective and efficient across tasks and languages, it remains difficult to create effective fully zero-shot dense retrieval systems when no re…
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer
Zhengbao Jiang, Luyu Gao, Jun Araki +4
Systems for knowledge-intensive tasks such as open-domain question answering (QA) usually consist of two stages: efficient retrieval of relevant documents from a large corpus and d…
PAL: Program-aided Language Models
Luyu Gao, Aman Madaan, Shuyan Zhou +5
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("f…
RARR: Researching and Revising What Language Models Say, Using Language Models
Luyu Gao, Zhuyun Dai, Panupong Pasupat +8
Language models (LMs) now excel at many tasks such as few-shot learning, question answering, reasoning, and dialog. However, they sometimes generate unsupported or misleading conte…
Long Document Re-ranking with Modular Re-ranker
Luyu Gao, Jamie Callan
Long document re-ranking has been a challenging problem for neural re-rankers based on deep language models like BERT. Early work breaks the documents into short passage-like chunk…
Tevatron: An Efficient and Flexible Toolkit for Dense Retrieval
Luyu Gao, Xueguang Ma, Jimmy Lin +1
Recent rapid advancements in deep pre-trained language models and the introductions of large datasets have powered research in embedding-based dense retrieval. While several good r…
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
Luyu Gao, Jamie Callan
Recent research demonstrates the effectiveness of using fine-tuned language models~(LM) for dense retrieval. However, dense retrievers are hard to train, typically requiring heavil…
COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
Luyu Gao, Zhuyun Dai, Jamie Callan
Classical information retrieval systems such as BM25 rely on exact lexical match and carry out search efficiently with inverted list index. Recent neural IR models shifts towards s…
Condenser: a Pre-training Architecture for Dense Retrieval
Luyu Gao, Jamie Callan
Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passa…
Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
Luyu Gao, Zhuyun Dai, Jamie Callan
Pre-trained deep language models~(LM) have advanced the state-of-the-art of text retrieval. Rerankers fine-tuned from deep LM estimates candidate relevance based on rich contextual…
Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
Luyu Gao, Yunyi Zhang, Jiawei Han +1
Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from…
Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation
Luyu Gao, Xinyi Wang, Graham Neubig
To improve the performance of Neural Machine Translation~(NMT) for low-resource languages~(LRL), one effective strategy is to leverage parallel data from a related high-resource la…
Understanding BERT Rankers Under Distillation
Luyu Gao, Zhuyun Dai, Jamie Callan
Deep language models such as BERT pre-trained on large corpus have given a huge performance boost to the state-of-the-art information retrieval ranking systems. Knowledge embedded…
Complementing Lexical Retrieval with Semantic Residual Embedding
Luyu Gao, Zhuyun Dai, Tongfei Chen +3
This paper presents CLEAR, a retrieval model that seeks to complement classical lexical exact-match models such as BM25 with semantic matching signals from a neural embedding match…
Modularized Transfomer-based Ranking Framework
Luyu Gao, Zhuyun Dai, Jamie Callan
Recent innovations in Transformer-based ranking models have advanced the state-of-the-art in information retrieval. However, these Transformers are computationally expensive, and t…