Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers

Abstract

This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar-sized baselines on prompted NLP benchmarks, such as T0 Eval and MMLU, and rivals the state-of-the-art T0-11B model with only 8% of its parameters. Our analysis on model's neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at https://github.com/gonglinyuan/metro_t0.

arXiv commentPublished as a conference paper at ACL 2023. 9 pages

References in corpus (15)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Scaling Instruction-Finetuned Language Models
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• A Simple Method for Commonsense Reasoning
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• Corrupted Image Modeling for Self-Supervised Visual Pre-Training
• METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
• Your Transformer May Not be as Powerful as You Expect
• MC-BERT: Efficient Language Pre-Training via a Meta Controller
• No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
• Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
• Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories