213 citations · 889 across the 55 of their papers we have counts for
56 papers
Fractional State Space Transition for Long Sequence Modeling
Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi +2
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Mos…
Gradually Excavating External Knowledge for Implicit Complex Question Answering
Chang Liu, Xiaoguang Li, Lifeng Shang +4
Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-…
Visually Guided Generative Text-Layout Pre-training for Document Intelligence
Zhiming Mao, Haoli Bai, Lu Hou +4
Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and r…
Preparing Lessons for Progressive Training on Language Models
Yu Pan, Ye Yuan, Yichun Yin +6
The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes. Prior…
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models
Haochen Tan, Zhijiang Guo, Zhan Shi +8
Large Language Models (LLMs) have succeeded remarkably in understanding long-form contents. However, exploring their capability for generating long-form contents, such as reports a…
Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment
Boyang Xue, Weichao Wang, Hongru Wang +7
Pretrained language models (PLMs) based knowledge-grounded dialogue systems are prone to generate responses that are factually inconsistent with the provided knowledge source. In s…
Reusing Pretrained Models by Multi-linear Operators for Efficient Training
Yu Pan, Ye Yuan, Yichun Yin +4
Training large models from scratch usually costs a substantial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained mod…
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
Yuxin Jiang, Yufei Wang, Xingshan Zeng +7
The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure res…
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models
Wai-Chung Kwan, Xingshan Zeng, Yufei Wang +5
Managing long sequences has become an important and necessary feature for large language models (LLMs). However, it is still an open question of how to comprehensively and systemat…
Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
Kai Chen, Chunwei Wang, Kuo Yang +11
The rapid development of large language models (LLMs) has not only provided numerous opportunities but also presented significant challenges. This becomes particularly evident when…
Prompt-Based Length Controlled Generation with Reinforcement Learning
Renlong Jie, Xiaojun Meng, Lifeng Shang +2
Large language models (LLMs) like ChatGPT and GPT-4 have attracted great attention given their surprising performance on a wide range of NLP tasks. Length controlled generation of…
AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models
Siheng Li, Cheng Yang, Yichun Yin +6
Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years. However, the research is still stym…
NewsDialogues: Towards Proactive News Grounded Conversation
Siheng Li, Yichun Yin, Cheng Yang +7
Hot news is one of the most popular topics in daily conversations. However, news grounded conversation has long been stymied by the lack of well-designed task definition and scarce…
Enhancing Coherence of Extractive Summarization with Multitask Learning
Renlong Jie, Xiaojun Meng, Lifeng Shang +2
This study proposes a multitask learning architecture for extractive summarization with coherence boosting. The architecture contains an extractive summarizer and coherent discrimi…
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation
Zhexin Zhang, Jiale Cheng, Hao Sun +5
Large pretrained language models can easily produce toxic or biased content, which is prohibitive for practical use. In order to detect such toxic generations, existing methods rel…
Retrieval-based Disentangled Representation Learning with Natural Language Supervision
Jiawei Zhou, Xiaoguang Li, Lifeng Shang +3
Disentangled representation learning remains challenging as the underlying factors of variation in the data do not naturally exist. The inherent complexity of real-world data makes…
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Zhongwei Wan, Yichun Yin, Wei Zhang +5
Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e.g., biomedical and computer science) by continuing to pre-train general PLMs…
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Dongsheng Chen, Chaofan Tao, Lu Hou +3
Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive du…
Pre-training Language Models with Deterministic Factual Knowledge
Shaobo Li, Xiaoguang Li, Lifeng Shang +5
Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge. However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitiv…
Exploring Extreme Parameter Compression for Pre-trained Language Models
Yuxin Ren, Benyou Wang, Lifeng Shang +2
Recent work explored the potential of large-scale Transformer-based pre-trained models, especially Pre-trained Language Models (PLMs) in natural language processing. This raises ma…
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering
Jiawei Zhou, Xiaoguang Li, Lifeng Shang +10
To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However,…
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
Shaobo Li, Xiaoguang Li, Lifeng Shang +6
Recently, there has been a trend to investigate the factual knowledge captured by Pre-trained Language Models (PLMs). Many works show the PLMs' ability to fill in the missing factu…
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Wenliang Dai, Lu Hou, Lifeng Shang +3
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on v…
PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
Fei Mi, Yitong Li, Yulong Zeng +7
In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021).…