1.5k citations · 5.2k across the 127 of their papers we have counts for
19 papers · 1 filter
Towards More Effective and Economic Sparsely-Activated Model
Hao Jiang, Ke Zhan, Jianwei Qu +14
The sparsely-activated models have achieved great success in natural language processing through large-scale parameters and relatively low computational cost, and gradually become…
KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier
Linyang Li, Demin Song, Ruotian Ma +2
Pre-trained models are widely used in fine-tuning downstream tasks with linear classifiers optimized by the cross-entropy loss, which might face robustness and stability problems.…
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline
Xiangyang Liu, Tianxiang Sun, Junliang He +7
Supersized pre-trained language models have pushed the accuracy of various natural language processing (NLP) tasks to a new state-of-the-art (SOTA). Rather than pursuing the reachl…
Learning to Teach with Student Feedback
Yitao Liu, Tianxiang Sun, Xipeng Qiu +1
Knowledge distillation (KD) has gained much attention due to its effectiveness in compressing large-scale pre-trained models. In typical KD methods, the small student model is trai…
Paradigm Shift in Natural Language Processing
Tianxiang Sun, Xiangyang Liu, Xipeng Qiu +1
In the era of deep learning, modeling for most NLP tasks has converged to several mainstream paradigms. For example, we usually adopt the sequence labeling paradigm to solve a bund…
RetrievalSum: A Retrieval Enhanced Framework for Abstractive Summarization
Chenxin An, Ming Zhong, Zhichao Geng +2
Existing summarization systems mostly generate summaries purely relying on the content of the source document. However, even for humans, we usually need some references or exemplar…
CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
Yunfan Shao, Zhichao Geng, Yitao Liu +6
In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTM…
Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning
Linyang Li, Demin Song, Xiaonan Li +3
\textbf{P}re-\textbf{T}rained \textbf{M}odel\textbf{s} have been widely applied and recently proved vulnerable under backdoor attacks: the released pre-trained weights can be malic…
A Survey of Transformers
Tianyang Lin, Yuxin Wang, Xiangyang Liu +1
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natura…
Accelerating BERT Inference for Sequence Labeling via Early-Exit
Xiaonan Li, Yunfan Shao, Tianxiang Sun +3
Both performance and efficiency are crucial factors for sequence labeling tasks in many real-world scenarios. Although the pre-trained models (PTMs) have significantly improved the…
A Unified Generative Framework for Aspect-Based Sentiment Analysis
Hang Yan, Junqi Dai, Tuo ji +2
Aspect-based Sentiment Analysis (ABSA) aims to identify the aspect terms, their corresponding sentiment polarities, and the opinion terms. There exist seven subtasks in ABSA. Most…
A Unified Generative Framework for Various NER Subtasks
Hang Yan, Tao Gui, Junqi Dai +3
Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be cate…
Pre-Trained Models: Past, Present and Future
Xu Han, Zhengyan Zhang, Ning Ding +21
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophis…
Early Exiting with Ensemble Internal Classifiers
Tianxiang Sun, Yunhua Zhou, Xiangyang Liu +5
As a simple technique to accelerate inference of large-scale pre-trained models, early exiting has gained much attention in the NLP community. It allows samples to exit early at in…
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Ming Zhong, Da Yin, Tao Yu +8
Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may…
Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa
Junqi Dai, Hang Yan, Tianxiang Sun +2
Aspect-based Sentiment Analysis (ABSA), aiming at predicting the polarities for aspects, is a fine-grained task in the field of sentiment analysis. Previous work showed syntactic i…
Enhancing Scientific Papers Summarization with Citation Graph
Chenxin An, Ming Zhong, Yiran Chen +3
Previous work for text summarization in scientific domain mainly focused on the content of the input document, but seldom considering its citation network. However, scientific pape…
Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning
Yichao Luo, Yige Xu, Jiacheng Ye +2
Aiming to generate a set of keyphrases, Keyphrase Generation (KG) is a classical task for capturing the central idea from a given document. Based on Seq2Seq models, the previous re…
Fork or Fail: Cycle-Consistent Training with Many-to-One Mappings
Qipeng Guo, Zhijing Jin, Ziyu Wang +5
Cycle-consistent training is widely used for jointly learning a forward and inverse mapping between two domains of interest without the cumbersome requirement of collecting matched…