100 citations · 183 across the 23 of their papers we have counts for
23 papers
Craw4LLM: Efficient Web Crawling for LLM Pretraining
Shi Yu, Zhiyuan Liu, Chenyan Xiong
Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper…
COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis
Weiqing Yang, Hanbin Wang, Zhenghao Liu +7
Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human d…
Cleaner Pretraining Corpus Curation with Neural Web Scraping
Zhipeng Xu, Zhenghao Liu, Yukun Yan +3
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation,…
Enhancing Dense Retrievers' Robustness with Group-level Reweighting
Peixuan Han, Zhenghao Liu, Zhiyuan Liu +1
The anchor-document data derived from web graphs offers a wealth of paired information for training dense retrieval models in an unsupervised manner. However, unsupervised data con…
MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin
Tianshuo Zhou, Sen Mei, Xinze Li +5
This paper proposes Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL), which learns an embedding space for queries and multi-modal documents to conduct retrieval. MARVE…
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model
Cheng Qian, Chenyan Xiong, Zhenghao Liu +1
Large Language Models (LLMs) have demonstrated remarkable progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptabilit…
Text Matching Improves Sequential Recommendation by Reducing Popularity Biases
Zhenghao Liu, Sen Mei, Chenyan Xiong +5
This paper proposes Text mAtching based SequenTial rEcommendation model (TASTE), which maps items and users in an embedding space and recommends items by matching their text repres…
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data
Xinze Li, Zhenghao Liu, Chenyan Xiong +4
This paper presents Structure Aware Dense Retrieval (SANTA) model, which encodes user queries and structured data in one universal embedding space for retrieving structured data. S…
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In
Zichun Yu, Chenyan Xiong, Shi Yu +1
Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly…
Fusion-in-T5: Unifying Document Ranking Signals for Improved Information Retrieval
Shi Yu, Chenghao Fan, Chenyan Xiong +3
Common document ranking pipelines in search systems are cascade systems that involve multiple ranking layers to integrate different information step-by-step. In this paper, we prop…
Rethinking Dense Retrieval's Few-Shot Ability
Si Sun, Yida Lu, Shi Yu +6
Few-shot dense retrieval (DR) aims to effectively generalize to novel search scenarios by learning a few samples. Despite its importance, there is little study on specialized datas…
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Si Sun, Chenyan Xiong, Yue Yu +3
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained mo…
Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv +2
This paper presents Universal Vision-Language Dense Retrieval (UniVL-DR), which builds a unified model for multi-modal retrieval. UniVL-DR encodes queries and multi-modality resour…
Improving Task Generalization via Unified Schema Prompt
Wanjun Zhong, Yifan Gao, Ning Ding +5
Task generalization has been a long standing challenge in Natural Language Processing (NLP). Recent research attempts to improve the task generalization ability of pre-trained lang…
P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
Xiaomeng Hu, Shi Yu, Chenyan Xiong +3
Compared to other language tasks, applying pre-trained language models (PLMs) for search ranking often requires more nuances and training signals. In this paper, we identify and st…
Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder
Zhenghao Liu, Han Zhang, Chenyan Xiong +3
Dense retrievers encode queries and documents and map them in an embedding space using pre-trained language models. These embeddings need to be high-dimensional to fit training sig…
Prototypical Verbalizer for Prompt-based Few-shot Tuning
Ganqu Cui, Shengding Hu, Ning Ding +2
Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze questi…
TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling
Huiyuan Xie, Zhenghao Liu, Chenyan Xiong +2
Human conversations naturally evolve around different topics and fluently move between them. In research on dialog systems, the ability to actively and smoothly transition to new t…
PPT: Pre-trained Prompt Tuning for Few-shot Learning
Yuxian Gu, Xu Han, Zhiyuan Liu +1
Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, p…
Prompt-Learning for Fine-Grained Entity Typing
Ning Ding, Yulin Chen, Xu Han +6
As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{clo…
More Robust Dense Retrieval with Contrastive Dual Learning
Yizhi Li, Zhenghao Liu, Chenyan Xiong +1
Dense retrieval conducts text retrieval in the embedding space and has shown many advantages compared to sparse retrieval. Existing dense retrievers optimize representations of que…
Capturing Global Informativeness in Open Domain Keyphrase Extraction
Si Sun, Zhenghao Liu, Chenyan Xiong +2
Open-domain KeyPhrase Extraction (KPE) aims to extract keyphrases from documents without domain or quality restrictions, e.g., web pages with variant domains and qualities. Recentl…
Block Belief Propagation for Parameter Learning in Markov Random Fields
You Lu, Zhiyuan Liu, Bert Huang
Traditional learning methods for training Markov random fields require doing inference over all variables to compute the likelihood gradient. The iteration complexity for those met…