221 citations · 739 across the 26 of their papers we have counts for
20 papers · 1 filter
Augmenting Language Models with Long-Term Memory
Weizhi Wang, Li Dong, Hao Cheng +4
Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs.…
Chain-of-Skills: A Configurable Model for Open-domain Question Answering
Kaixin Ma, Hao Cheng, Yu Zhang +3
The retrieval model is an indispensable component for real-world knowledge-intensive tasks, e.g., open-domain question answering (ODQA). As separate retrieval skills are annotated…
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Linyuan Gong, Chenyan Xiong, Xiaodong Liu +5
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain…
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Yu Zhang, Hao Cheng, Zhihong Shen +3
Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pre-trained language models (LMs) have shown…
Pre-training Transformers for Knowledge Graph Completion
Sanxing Chen, Hao Cheng, Xiaodong Liu +3
Learning transferable representation of knowledge graphs (KGs) is challenging due to the heterogeneous, multi-relational nature of graph structures. Inspired by Transformer-based p…
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee +4
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of…