299 citations · 942 across the 23 of their papers we have counts for
10 papers · 1 filter
Weight-Inherited Distillation for Task-Agnostic BERT Compression
Taiqiang Wu, Cheng Hou, Shanshan Lao +4
Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the b…
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning
Kunbo Ding, Weijie Liu, Yuejian Fang +6
Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage f…
HyperPrompt: Prompt-based Task-Conditioning of Transformers
Yun He, Huaixiu Steven Zheng, Yi Tay +9
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way. Here, we explore the use of HyperNetworks to generate hyper-prompts: we pro…
Semantic Matching from Different Perspectives
Weijie Liu, Tao Zhu, Weiquan Mao +4
In this paper, we pay attention to the issue which is usually overlooked, i.e., \textit{similarity should be determined from different perspectives}. To explore this issue, we rele…
Transformer Memory as a Differentiable Search Index
Yi Tay, Vinh Q. Tran, Mostafa Dehghani +10
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of th…
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
Yi Tay, Zhe Zhao, Dara Bahri +2
Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a…
Synthesizer: Rethinking Self-Attention in Transformer Models
Yi Tay, Dara Bahri, Donald Metzler +3
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importan…
FastBERT: a Self-distilling BERT with Adaptive Inference Time
Weijie Liu, Peng Zhou, Zhe Zhao +3
Pre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can…
K-BERT: Enabling Language Representation with Knowledge Graph
Weijie Liu, Peng Zhou, Zhe Zhao +4
Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a dom…
UER: An Open-Source Toolkit for Pre-training Models
Zhe Zhao, Hui Chen, Jinbin Zhang +7
Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks. While there does not exist a single pre-training model that works best in all c…