90 citations · 112 across the 12 of their papers we have counts for
12 papers
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Shicheng Tan, Weng Lam Tam, Yuanchun Wang +9
Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment…
Task-agnostic Distillation of Encoder-Decoder Language Models
Chen Zhang, Yang Yang, Jingang Wang +1
Finetuning pretrained language models (LMs) have enabled appealing performance on a diverse array of tasks. The intriguing task-agnostic property has driven a shifted focus from ta…
Lifting the Curse of Capacity Gap in Distilling Language Models
Chen Zhang, Yang Yang, Jiahao Liu +4
Pretrained language models (LMs) have shown compelling performance on various downstream tasks, but unfortunately they require a tremendous amount of inference compute. Knowledge d…
Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration
Liqi Yan, Qifan Wang, Siqi Ma +2
Instance segmentation in videos, which aims to segment and track multiple objects in video frames, has garnered a flurry of research attention in recent years. In this paper, we pr…
XPrompt: Exploring the Extreme of Prompt Tuning
Fang Ma, Chen Zhang, Lei Ren +5
Prompt tuning learns soft prompts to condition frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner. While prompt tuning has gr…
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations
Borun Chen, Hongyin Tang, Jiahao Bu +6
Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been suc…
Structural Bias for Aspect Sentiment Triplet Extraction
Chen Zhang, Lei Ren, Fang Ma +3
Structural bias has recently been exploited for aspect sentiment triplet extraction (ASTE) and led to improved performance. On the other hand, it is recognized that explicitly inco…
MiniDisc: Minimal Distillation Schedule for Language Model Compression
Chen Zhang, Yang Yang, Qifan Wang +4
Recent studies have uncovered that language model distillation is less effective when facing a large capacity gap between the teacher and the student, and introduced teacher assist…
Making Pretrained Language Models Good Long-tailed Learners
Chen Zhang, Lei Ren, Jingang Wang +2
Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the…
GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval
Jiduan Liu, Jiahao Liu, Yang Yang +4
Recently, retrieval models based on dense representations are dominant in passage retrieval tasks, due to their outstanding ability in terms of capturing semantics of input text co…
VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction
Dan Li, Yang Yang, Hongyin Tang +4
With the booming of pre-trained transformers, representation-based models based on Siamese transformer encoders have become mainstream techniques for efficient text matching. Howev…
Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
Hongyin Tang, Xingwu Sun, Beihong Jin +3
Recently, the retrieval models based on dense representations have been gradually applied in the first stage of the document retrieval tasks, showing better performance than tradit…