28 citations · 47 across the 18 of their papers we have counts for
6 papers · 1 filter
On Elastic Language Models
Chen Zhang, Benyou Wang, Dawei Song
Large-scale pretrained language models have achieved compelling performance in a wide range of language understanding and information retrieval tasks. Knowledge distillation offers…
Sparse Contrastive Learning of Sentence Embeddings
Ruize An, Chen Zhang, Dawei Song
Recently, SimCSE has shown the feasibility of contrastive learning in training sentence embeddings and illustrates its expressiveness in spanning an aligned and uniform embedding s…
Towards the Law of Capacity Gap in Distilling Language Models
Chen Zhang, Qiuchi Li, Dawei Song +3
Language model (LM) distillation aims at distilling the knowledge in a large teacher LM to a small student one. As a critical issue facing LM distillation, a superior student often…
Task-agnostic Distillation of Encoder-Decoder Language Models
Chen Zhang, Yang Yang, Jingang Wang +1
Finetuning pretrained language models (LMs) have enabled appealing performance on a diverse array of tasks. The intriguing task-agnostic property has driven a shifted focus from ta…
Lifting the Curse of Capacity Gap in Distilling Language Models
Chen Zhang, Yang Yang, Jiahao Liu +4
Pretrained language models (LMs) have shown compelling performance on various downstream tasks, but unfortunately they require a tremendous amount of inference compute. Knowledge d…
Modular Retrieval for Generalization and Interpretation
Juhao Liang, Chen Zhang, Zhengyang Tang +3
New retrieval tasks have always been emerging, thus urging the development of new retrieval models. However, instantiating a retrieval model for each new retrieval task is resource…