4 citations · 9 across the 17 of their papers we have counts for
5 papers · 1 filter
The Efficiency Gap in Byte Modeling
Celine Lee, Jing Nathan Yan, Chen Liang +9
Modern language models have historically relied on two dominant design choices: subword tokenization and autoregressive (AR) ordering. These design decisions bake in priors that di…
ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation
Chengyu Dong, Huan Gui, Noveen Sachdeva +6
Knowledge distillation from pretrained visual representation models offers an effective approach to improve small, task-specific production models. However, the effectiveness of su…
How to Train Data-Efficient LLMs
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang +6
The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto…
Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts
Zichang Liu, Qingyun Liu, Yuening Li +6
Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality. For example, in our experimen…
LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views
Yuji Roh, Qingyun Liu, Huan Gui +8
Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various task…