Task Compass: Scaling Multi-task Pre-training with Task Prefix

Zhuosheng Zhang, Shuohang Wang, Yichong Xu +6

Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language mod…