213 citations · 889 across the 55 of their papers we have counts for
13 papers · 1 filter
bert2BERT: Towards Reusable Pretrained Language Models
Cheng Chen, Yichun Yin, Lifeng Shang +7
In recent years, researchers tend to pre-train ever-larger language models to explore the upper limit of deep models. However, large language model pre-training costs intensive com…
Towards Efficient Post-training Quantization of Pre-trained Language Models
Haoli Bai, Lu Hou, Lifeng Shang +3
Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow…
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Chenyang Lyu, Lifeng Shang, Yvette Graham +3
Question Generation (QG) is the task of generating a plausible question for a given <passage, answer> pair. Template-based QG uses linguistically-informed heuristics to transform d…
Generate & Rank: A Multi-task Framework for Math Word Problems
Jianhao Shen, Yichun Yin, Lin Li +4
Math word problem (MWP) is a challenging and critical task in natural language processing. Many recent studies formalize MWP as a generation task and have adopted sequence-to-seque…
Integrating Regular Expressions with Neural Networks via DFA
Shaobo Li, Qun Liu, Xin Jiang +5
Human-designed rules are widely used to build industry applications. However, it is infeasible to maintain thousands of such hand-crafted rules. So it is very important to integrat…
DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling
Baojun Wang, Zhao Zhang, Kun Xu +7
Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks. However, previous works commonly have difficulty dealing…
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Yichun Yin, Cheng Chen, Lifeng Shang +3
Pre-trained language models (PLMs) have achieved great success in natural language processing. Most of PLMs follow the default setting of architecture hyper-parameters (e.g., the h…
A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
Zhihong Shao, Lifeng Shang, Qun Liu +1
Weakly supervised question answering usually has only the final answers as supervision signals while the correct solutions to derive the answers are not provided. This setting give…
Improved OOD Generalization via Adversarial Training and Pre-training
Mingyang Yi, Lu Hou, Jiacheng Sun +4
Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OO…
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Cheng Chen, Yichun Yin, Lifeng Shang +4
Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression. Although achieving promising results on NLP tasks, it requires en…
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Mingyang Yi, Lu Hou, Lifeng Shang +3
Data augmentation is an effective technique to improve the generalization of deep neural networks. However, previous data augmentation methods usually treat the augmented samples e…
LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
Xiaoqi Jiao, Yichun Yin, Lifeng Shang +5
The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these mod…
Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation
Chang Liu, Xiaoguang Li, Guohao Cai +3
Sequential recommender systems aim to model users' evolving interests from their historical behaviors, and hence make customized time-relevant recommendations. Compared with tradit…