14 citations · 14 across the 6 of their papers we have counts for
6 papers
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
Yuanyi Wang, Su Lu, Yanggan Gu +6
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by pr…
GLM-Dialog: Noise-tolerant Pre-training for Knowledge-grounded Dialogue Generation
Jing Zhang, Xiaokang Zhang, Daniel Zhang-Li +10
We present GLM-Dialog, a large-scale language model (LLM) with 10B parameters capable of knowledge-grounded conversation in Chinese using a search engine to access the Internet kno…
Generalized Knowledge Distillation via Relationship Matching
Han-Jia Ye, Su Lu, De-Chuan Zhan
The knowledge of a well-trained deep neural network (a.k.a. the "teacher") is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and in…
Selective Cross-Task Distillation
Su Lu, Han-Jia Ye, De-Chuan Zhan
The outpouring of various pre-trained models empowers knowledge distillation by providing abundant teacher resources, but there lacks a developed mechanism to utilize these teacher…
Few-Shot Action Recognition with Compromised Metric via Optimal Transport
Su Lu, Han-Jia Ye, De-Chuan Zhan
Although vital to computer vision systems, few-shot action recognition is still not mature despite the wide research of few-shot image classification. Popular few-shot learning alg…
Towards Enabling Meta-Learning from Target Models
Su Lu, Han-Jia Ye, Le Gan +1
Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. It is often realized through optimizing a meta-model with the ev…