Pretrained Transformers as Universal Computation Engines

Abstract

We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks. We consider such a model, which we call a Frozen Pretrained Transformer (FPT), and study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. In contrast to prior works which investigate finetuning on the same modality as the pretraining dataset, we show that pretraining on natural language can improve performance and compute efficiency on non-language downstream tasks. Additionally, we perform an analysis of the architecture, comparing the performance of a random initialized transformer to a random LSTM. Combining the two insights, we find language-pretrained transformers can obtain strong performance on a variety of non-language tasks.

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Learning Transferable Visual Models From Natural Language Supervision
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Recent Advances in Physical Reservoir Computing: A Review
• Adversarial Feature Learning
• On the Cross-lingual Transferability of Monolingual Representations
• Learning multiple visual domains with residual adapters
• One Model To Learn Them All
• Deep Equilibrium Models
• Long Range Arena: A Benchmark for Efficient Transformers
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Evaluating Protein Transfer Learning with TAPE
• Hopfield Networks is All You Need
• On Self Modulation for Generative Adversarial Networks
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Imitating Interactive Intelligence
• Scaling Laws for Transfer
• LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning

Cited by in corpus (25)

• On the Opportunities and Risks of Foundation Models
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Multimodal Few-Shot Learning with Frozen Language Models
• StorSeismic: A new paradigm in deep learning for seismic processing
• Choose a Transformer: Fourier or Galerkin
• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach
• FoodSAM: Any Food Segmentation
• Discrete and continuous representations and processing in deep learning: Looking forward
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• Exploring and Evaluating Personalized Models for Code Generation
• Robust Optimization for Multilingual Translation with Imbalanced Data
• What is Wrong with Language Models that Can Not Tell a Story?
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• On the ability of monolingual models to learn language-agnostic representations
• Compositional generalization in semantic parsing with pretrained transformers
• Pretrained Language Models are Symbolic Mathematics Solvers too!
• Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers
• Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers
• Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
• Generalization over different cellular automata rules learned by a deep feed-forward neural network