19 citations · 64 across the 25 of their papers we have counts for
26 papers
LISRec: Modeling User Preferences with Learned Item Shortcuts for Sequential Recommendation
Haidong Xin, Zhenghao Liu, Sen Mei +7
User-item interaction histories are pivotal for sequential recommendation systems but often include noise, such as unintended clicks or actions that fail to reflect genuine user pr…
ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation
Pengcheng Huang, Zhenghao Liu, Yukun Yan +8
Large language models (LLMs) integrated with retrieval-augmented generation (RAG) have improved factuality by grounding outputs in external evidence. However, they remain susceptib…
Craw4LLM: Efficient Web Crawling for LLM Pretraining
Shi Yu, Zhiyuan Liu, Chenyan Xiong
Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper…
RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
Xinze Li, Sen Mei, Zhenghao Liu +9
Retrieval-Augmented Generation (RAG) has proven its effectiveness in mitigating hallucinations in Large Language Models (LLMs) by retrieving knowledge from external resources. To a…
Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression
Xinze Li, Zhenghao Liu, Chenyan Xiong +4
Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In thi…
Cleaner Pretraining Corpus Curation with Neural Web Scraping
Zhipeng Xu, Zhenghao Liu, Yukun Yan +3
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation,…
ThinkNote: Enhancing Knowledge Integration and Utilization of Large Language Models via Constructivist Cognition Modeling
Zhipeng Xu, Zhenghao Liu, Yukun Yan +7
Large Language Models (LLMs) have demonstrated strong performance across a wide range of NLP tasks. However, they often exhibit suboptimal behaviors and inconsistencies when expose…
Enhancing Dense Retrievers' Robustness with Group-level Reweighting
Peixuan Han, Zhenghao Liu, Zhiyuan Liu +1
The anchor-document data derived from web graphs offers a wealth of paired information for training dense retrieval models in an unsupervised manner. However, unsupervised data con…
MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin
Tianshuo Zhou, Sen Mei, Xinze Li +5
This paper proposes Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL), which learns an embedding space for queries and multi-modal documents to conduct retrieval. MARVE…
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model
Cheng Qian, Chenyan Xiong, Zhenghao Liu +1
Large Language Models (LLMs) have demonstrated remarkable progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptabilit…
Improving Multitask Retrieval by Promoting Task Specialization
Wenzheng Zhang, Chenyan Xiong, Karl Stratos +1
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-spe…
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data
Xinze Li, Zhenghao Liu, Chenyan Xiong +4
This paper presents Structure Aware Dense Retrieval (SANTA) model, which encodes user queries and structured data in one universal embedding space for retrieving structured data. S…
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In
Zichun Yu, Chenyan Xiong, Shi Yu +1
Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly…
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Linyuan Gong, Chenyan Xiong, Xiaodong Liu +5
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain…
Unsupervised Dense Retrieval Training with Web Anchors
Yiqing Xie, Xiao Liu, Chenyan Xiong
In this work, we present an unsupervised retrieval method with contrastive learning on web anchors. The anchor text describes the content that is referenced from the linked page. T…
Fusion-in-T5: Unifying Document Ranking Signals for Improved Information Retrieval
Shi Yu, Chenghao Fan, Chenyan Xiong +3
Common document ranking pipelines in search systems are cascade systems that involve multiple ranking layers to integrate different information step-by-step. In this paper, we prop…
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
Suyu Ge, Chenyan Xiong, Corby Rosset +3
In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from m…
ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information
Arnold Overwijk, Chenyan Xiong, Xiao Liu +2
ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high qu…
COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
Yue Yu, Chenyan Xiong, Si Sun +2
We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source tr…
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Si Sun, Chenyan Xiong, Yue Yu +3
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained mo…
Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv +2
This paper presents Universal Vision-Language Dense Retrieval (UniVL-DR), which builds a unified model for multi-modal retrieval. UniVL-DR encodes queries and multi-modality resour…
Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder
Zhenghao Liu, Han Zhang, Chenyan Xiong +3
Dense retrievers encode queries and documents and map them in an embedding space using pre-trained language models. These embeddings need to be high-dimensional to fit training sig…
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
Payal Bajaj, Chenyan Xiong, Guolin Ke +7
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training…
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
Yu Meng, Chenyan Xiong, Payal Bajaj +4
We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-…