GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Abstract

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

Accepted to ICML 2022

Cited by in corpus (15)

• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
• Learning from models beyond fine-tuning
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Enhanced Automated Code Vulnerability Repair using Large Language Models
• From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape
• FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
• Transforming Visual Scene Graphs to Image Captions
• Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function
• Modularity in Deep Learning: A Survey
• Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?
• Exploring the Impact of Model Scaling on Parameter-Efficient Tuning
• COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search
• Cache Management for Mixture-of-Experts LLMs -- extended version
• Emergent inabilities? Inverse scaling over the course of pretraining
• Political Bias in LLMs: Unaligned Moral Values in Agent-centric Simulations