4 citations · 11 across the 19 of their papers we have counts for
4 papers · 1 filter
ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation
Chengyu Dong, Huan Gui, Noveen Sachdeva +6
Knowledge distillation from pretrained visual representation models offers an effective approach to improve small, task-specific production models. However, the effectiveness of su…
How to Train Data-Efficient LLMs
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang +6
The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto…
Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts
Zichang Liu, Qingyun Liu, Yuening Li +6
Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality. For example, in our experimen…
LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views
Yuji Roh, Qingyun Liu, Huan Gui +8
Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various task…