AdapterDistillation: Non-Destructive Task Composition with Knowledge Distillation

Junjie Wang, Yicheng Chen, Wangshu Zhang +3

Leveraging knowledge from multiple tasks through introducing a small number of task specific parameters into each transformer layer, also known as adapters, receives much attention…