1.4k citations · 5.4k across the 120 of their papers we have counts for
81 papers · 1 filter
QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation
Kunlun Zhu, Shihao Liang, Xu Han +4
Recent years have witnessed the success of question answering (QA), especially its potential to be a foundation paradigm for tackling diverse NLP tasks. However, obtaining sufficie…
CPM-2: Large-scale Cost-effective Pre-trained Language Models
Zhengyan Zhang, Yuxian Gu, Xu Han +16
In recent years, the size of pre-trained language models (PLMs) has grown by leaps and bounds. However, efficiency issues of these large-scale PLMs limit their utilization in real-…
Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution
Fanchao Qi, Yuan Yao, Sophia Xu +2
Recent studies show that neural natural language processing (NLP) models are vulnerable to backdoor attacks. Injected with backdoors, models perform normally on benign examples but…
Automatic Construction of Sememe Knowledge Bases via Dictionaries
Fanchao Qi, Yangyi Chen, Fengyu Wang +3
A sememe is defined as the minimum semantic unit in linguistics. Sememe knowledge bases (SKBs), which comprise words annotated with sememes, enable sememes to be applied to natural…
Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
Fanchao Qi, Mukai Li, Yangyi Chen +4
Backdoor attacks are a kind of insidious security threat against machine learning models. After being injected with a backdoor in training, the victim model will produce adversary-…
CLEVE: Contrastive Pre-training for Event Extraction
Ziqi Wang, Xiaozhi Wang, Xu Han +6
Event extraction (EE) has considerably benefited from pre-trained language models (PLMs) by fine-tuning. However, existing pre-training methods have not involved modeling event cha…
Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction
Tianyu Gao, Xu Han, Keyue Qiu +7
Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto-labeled data. However, its evaluation has l…
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu +2
Legal artificial intelligence (LegalAI) aims to benefit legal systems with the technology of artificial intelligence, especially natural language processing (NLP). Recently, inspir…
Knowledge Inheritance for Pre-trained Language Models
Yujia Qin, Yankai Lin, Jing Yi +8
Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PL…
PTR: Prompt Tuning with Rules for Text Classification
Xu Han, Weilin Zhao, Ning Ding +2
Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the…
Few-NERD: A Few-Shot Named Entity Recognition Dataset
Ning Ding, Guangwei Xu, Yulin Chen +5
Recently, considerable literature has grown up around the theme of few-shot named entity recognition (NER), but little published benchmark data specifically focused on the practica…
Equality before the Law: Legal Judgment Consistency Analysis for Fairness
Yuzhong Wang, Chaojun Xiao, Shirong Ma +5
In a legal system, judgment consistency is regarded as one of the most important manifestations of fairness. However, due to the complexity of factual elements that impact sentenci…
Representation Learning for Natural Language Processing
Zhiyuan Liu, Yankai Lin, Maosong Sun
This book aims to review and present the recent advances of distributed representation learning for NLP, including why representation learning can improve NLP, how representation l…
Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li +6
Pre-trained models (PTMs) have been widely used in various downstream tasks. The parameters of PTMs are distributed on the Internet and may suffer backdoor attacks. In this work, w…
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
Yujia Qin, Yankai Lin, Ryuichi Takanobu +6
Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do n…
CPM: A Large-scale Generative Chinese Pre-trained Language Model
Zhengyan Zhang, Xu Han, Hao Zhou +22
Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB training data, drew a lot o…
Learning from Context or Names? An Empirical Study on Neural Relation Extraction
Hao Peng, Tianyu Gao, Xu Han +5
Neural models have achieved remarkable success on relation extraction (RE) benchmarks. However, there is no clear understanding which type of information affects existing RE models…
Denoising Relation Extraction from Document-level Distant Supervision
Chaojun Xiao, Yuan Yao, Ruobing Xie +5
Distant supervision (DS) has been widely used to generate auto-labeled data for sentence-level relation extraction (RE), which improves RE performance. However, the existing succes…
Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu +2
Deep pre-trained Transformer models have achieved state-of-the-art results over a variety of natural language processing (NLP) tasks. By learning rich language knowledge with milli…
ONION: A Simple and Effective Defense Against Textual Backdoor Attacks
Fanchao Qi, Yangyi Chen, Mukai Li +3
Backdoor attacks are a kind of emergent training-time threat to deep neural networks (DNNs). They can manipulate the output of DNNs and possess high insidiousness. In the field of…
Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph
Xin Lv, Xu Han, Lei Hou +6
Multi-hop reasoning has been widely studied in recent years to seek an effective and interpretable method for knowledge graph (KG) completion. Most previous reasoning methods are d…
Partially-Aligned Data-to-Text Generation with Distant Supervision
Zihao Fu, Bei Shi, Wai Lam +2
The Data-to-Text task aims to generate human-readable text for describing some given structured data enabling more interpretability. However, the typical generation task is confine…
Learning to Attack: Towards Textual Adversarial Attacking in Real-world Situations
Yuan Zang, Bairu Hou, Fanchao Qi +3
Adversarial attacking aims to fool deep neural networks with adversarial examples. In the field of natural language processing, various textual adversarial attack models have been…
CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models
Yusheng Su, Xu Han, Zhengyan Zhang +5
Several recent efforts have been devoted to enhancing pre-trained language models (PLMs) by utilizing extra heterogeneous knowledge in knowledge graphs (KGs) and achieved consisten…