19 citations · 48 across the 14 of their papers we have counts for
4 papers · 1 filter
Co-training and Co-distillation for Quality Improvement and Compression of Language Models
Hayeon Lee, Rui Hou, Jongpil Kim +4
Knowledge Distillation (KD) compresses computationally expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models, allowing their use in resourc…
A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models
Hayeon Lee, Rui Hou, Jongpil Kim +3
Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance. Previous studi…
Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets
Hayeon Lee, Sohyun An, Minseon Kim +1
Distillation-aware Neural Architecture Search (DaNAS) aims to search for an optimal student architecture that obtains the best performance and/or efficiency when distilling the kno…
DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models
Sohyun An, Hayeon Lee, Jaehyeong Jo +2
Existing NAS methods suffer from either an excessive amount of time for repetitive sampling and training of many task-irrelevant architectures. To tackle such limitations of existi…