100 citations · 183 across the 23 of their papers we have counts for
12 papers · 1 filter
Craw4LLM: Efficient Web Crawling for LLM Pretraining
Shi Yu, Zhiyuan Liu, Chenyan Xiong
Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper…
Cleaner Pretraining Corpus Curation with Neural Web Scraping
Zhipeng Xu, Zhenghao Liu, Yukun Yan +3
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation,…
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model
Cheng Qian, Chenyan Xiong, Zhenghao Liu +1
Large Language Models (LLMs) have demonstrated remarkable progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptabilit…
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In
Zichun Yu, Chenyan Xiong, Shi Yu +1
Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly…
Rethinking Dense Retrieval's Few-Shot Ability
Si Sun, Yida Lu, Shi Yu +6
Few-shot dense retrieval (DR) aims to effectively generalize to novel search scenarios by learning a few samples. Despite its importance, there is little study on specialized datas…
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Si Sun, Chenyan Xiong, Yue Yu +3
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained mo…
Improving Task Generalization via Unified Schema Prompt
Wanjun Zhong, Yifan Gao, Ning Ding +5
Task generalization has been a long standing challenge in Natural Language Processing (NLP). Recent research attempts to improve the task generalization ability of pre-trained lang…
Prototypical Verbalizer for Prompt-based Few-shot Tuning
Ganqu Cui, Shengding Hu, Ning Ding +2
Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze questi…
TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling
Huiyuan Xie, Zhenghao Liu, Chenyan Xiong +2
Human conversations naturally evolve around different topics and fluently move between them. In research on dialog systems, the ability to actively and smoothly transition to new t…
PPT: Pre-trained Prompt Tuning for Few-shot Learning
Yuxian Gu, Xu Han, Zhiyuan Liu +1
Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, p…
Prompt-Learning for Fine-Grained Entity Typing
Ning Ding, Yulin Chen, Xu Han +6
As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{clo…
Capturing Global Informativeness in Open Domain Keyphrase Extraction
Si Sun, Zhenghao Liu, Chenyan Xiong +2
Open-domain KeyPhrase Extraction (KPE) aims to extract keyphrases from documents without domain or quality restrictions, e.g., web pages with variant domains and qualities. Recentl…