257 citations · 380 across the 46 of their papers we have counts for
4 papers · 1 filter
Gradient Knowledge Distillation for Pre-trained Language Models
Lean Wang, Lei Li, Xu Sun
Knowledge distillation (KD) is an effective framework to transfer knowledge from a large-scale teacher to a compact yet well-performing student. Previous KD practices for pre-train…
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models
Lei Li, Yankai Lin, Xuancheng Ren +4
Investigating better ways to reuse the released pre-trained language models (PLMs) can significantly reduce the computational cost and the potential environmental side-effects. Thi…
Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Prediction
Lei Li, Zhiyuan Zhang, Ruihan Bao +2
Traditional knowledge distillation in classification problems transfers the knowledge via class correlations in the soft label produced by teacher models, which are not available i…
Delving into the Openness of CLIP
Shuhuai Ren, Lei Li, Xuancheng Ren +2
Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descri…