What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

Thomas Wang, Adam Roberts, Daniel Hesslow +5

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly traine…