189 citations · 1.7k across the 110 of their papers we have counts for
6 papers · 1 filter
Gradient Knowledge Distillation for Pre-trained Language Models
Lean Wang, Lei Li, Xu Sun
Knowledge distillation (KD) is an effective framework to transfer knowledge from a large-scale teacher to a compact yet well-performing student. Previous KD practices for pre-train…
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models
Lei Li, Yankai Lin, Xuancheng Ren +4
Investigating better ways to reuse the released pre-trained language models (PLMs) can significantly reduce the computational cost and the potential environmental side-effects. Thi…
Prophet Attention: Predicting Attention with Future Attention for Image Captioning
Fenglin Liu, Xuancheng Ren, Xian Wu +3
Recently, attention based models have been used extensively in many sequence-to-sequence learning systems. Especially for image captioning, the attention based models are expected…
Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Prediction
Lei Li, Zhiyuan Zhang, Ruihan Bao +2
Traditional knowledge distillation in classification problems transfers the knowledge via class correlations in the soft label produced by teacher models, which are not available i…
Delving into the Openness of CLIP
Shuhuai Ren, Lei Li, Xuancheng Ren +2
Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descri…
Hierarchical Inductive Transfer for Continual Dialogue Learning
Shaoxiong Feng, Xuancheng Ren, Kan Li +1
Pre-trained models have achieved excellent performance on the dialogue task. However, for the continual increase of online chit-chat scenarios, directly fine-tuning these models fo…