Controlling Computation versus Quality for Neural Sequence Models

Abstract

Most neural networks utilize the same amount of compute for every example independent of the inherent complexity of the input. Further, methods that adapt the amount of computation to the example focus on finding a fixed inference-time computational graph per example, ignoring any external computational budgets or varying inference time limitations. In this work, we utilize conditional computation to make neural sequence models (Transformer) more efficient and computation-aware during inference. We first modify the Transformer architecture, making each set of operations conditionally executable depending on the output of a learned control network. We then train this model in a multi-task setting, where each task corresponds to a particular computation budget. This allows us to train a single model that can be controlled to operate on different points of the computation-quality trade-off curve, depending on the available computation budget at inference time. We evaluate our approach on two tasks: (i) WMT English-French Translation and (ii) Unsupervised representation learning (BERT). Our experiments demonstrate that the proposed Conditional Computation Transformer (CCT) is competitive with vanilla Transformers when allowed to utilize its full computational budget, while improving significantly over computationally equivalent baselines when operating on smaller computational budgets.

References in corpus (16)

• Sequence to Sequence Learning with Neural Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Universal Transformers
• Adaptive Computation Time for Recurrent Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Conditional Computation in Neural Networks for faster models
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Large Memory Layers with Product Keys
• Mixture Models for Diverse Machine Translation: Tricks of the Trade
• Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
• Monotonic Chunkwise Attention
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning

Cited by in corpus (6)

• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
• Surprisal-Triggered Conditional Computation with Neural Networks
• IOT: Instance-wise Layer Reordering for Transformer Structures
• FNetAR: Mixing Tokens with Autoregressive Fourier Transforms
• When in Doubt, Summon the Titans: Efficient Inference with Large Models