123 citations · 335 across the 26 of their papers we have counts for
27 papers
Thinking Augmented Pre-training
Liang Wang, Nan Yang, Shaohan Huang +2
This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectori…
Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang +3
This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Conventional R…
Bootstrap Your Own Context Length
Liang Wang, Nan Yang, Xingxing Zhang +2
We introduce a bootstrapping approach to train long-context language models by exploiting their short-context capabilities only. Our method utilizes a simple agent workflow to synt…
Little Giants: Synthesizing High-Quality Embedding Data at Scale
Haonan Chen, Liang Wang, Nan Yang +4
Synthetic data generation has become an increasingly popular way of training models without the need for large, manually labeled datasets. For tasks like text embedding, synthetic…
LongEmbed: Extending Embedding Models for Long Context Retrieval
Dawei Zhu, Liang Wang, Nan Yang +4
Embedding models play a pivot role in modern NLP applications such as IR and RAG. While the context limit of LLMs has been pushed beyond 1 million tokens, embedding models are stil…
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang +3
This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding model…
Generative Representational Instruction Tuning
Niklas Muennighoff, Hongjin Su, Liang Wang +5
All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational in…
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang +3
In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods…
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang +2
The effectiveness of multi-stage text retrieval has been solidly demonstrated since before the era of pre-trained language models. However, most existing studies utilize models tha…
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang +3
Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These compo…
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Dawei Zhu, Nan Yang, Liang Wang +4
Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer…
Learning to Retrieve In-Context Examples for Large Language Models
Liang Wang, Nan Yang, Furu Wei
Large language models (LLMs) have demonstrated their ability to learn in-context, allowing them to perform various tasks based on a few input-output examples. However, the effectiv…
Learning to Rank in Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang +2
Generative retrieval stands out as a promising new paradigm in text retrieval that aims to generate identifier strings of relevant passages as the retrieval target. This generative…
Multiview Identifiers Enhanced Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang +2
Instead of simply matching a query to pre-existing passages, generative retrieval generates identifier strings of passages as the retrieval target. At a cost, the identifier must b…
Inference with Reference: Lossless Acceleration of Large Language Models
Nan Yang, Tao Ge, Liang Wang +5
We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identi…
Query2doc: Query Expansion with Large Language Models
Liang Wang, Nan Yang, Furu Wei
This paper introduces a simple yet effective query expansion approach, denoted as query2doc, to improve both sparse and dense retrieval systems. The proposed method first generates…
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang +5
This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision…
Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
Zehan Li, Nan Yang, Liang Wang +1
In this paper, we propose a new dense retrieval model which learns diverse document representations with deep query interactions. Our model encodes each document with a set of gene…
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang, Nan Yang, Xiaolong Huang +5
In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple…
SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Liang Wang, Wei Zhao, Zhuoyu Wei +1
Knowledge graph completion (KGC) aims to reason over known facts and infer the missing links. Text-based methods such as KGBERT (Yao et al., 2019) learn entity representations from…
Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
Liang Wang, Wei Zhao, Jingming Liu
In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolingual…
Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems
Wei Zhao, Mingyue Shang, Yang Liu +2
Automatic math word problem solving has attracted growing attention in recent years. The evaluation datasets used by previous works have serious limitations in terms of scale and d…
Investigating Label Bias in Beam Search for Open-ended Text Generation
Liang Wang, Jinlong Liu, Jingming Liu
Beam search is an effective and widely used decoding algorithm in many sequence-to-sequence (seq2seq) text generation tasks. However, in open-ended text generation, beam search is…
Denoising based Sequence-to-Sequence Pre-training for Text Generation
Liang Wang, Wei Zhao, Ruoyu Jia +2
This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generatio…