45 citations · 260 across the 35 of their papers we have counts for
3 papers · 1 filter
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
Fuzhao Xue, Yao Fu, Wangchunshu Zhou +2
Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, a…
A Survey on Green Deep Learning
Jingjing Xu, Wangchunshu Zhou, Zhiyi Fu +2
In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and…
BERT Learns to Teach: Knowledge Distillation with Meta Learning
Wangchunshu Zhou, Canwen Xu, Julian McAuley
We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is…