The Evolved Transformer

Abstract

Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments -- the Evolved Transformer -- demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014 English-German, WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size, the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8 on WMT'14 English-German; at smaller sizes, it achieves the same quality as the original "big" Transformer with 37.6% less parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.

ICML version with SOTA results

Cited by in corpus (105)

• A Survey on Visual Transformer
• On the Opportunities and Risks of Foundation Models
• A Survey on Evolutionary Neural Architecture Search
• Towards a Human-like Open-Domain Chatbot
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• A Survey on Neural Architecture Search
• Incorporating BERT into Neural Machine Translation
• Evaluating the Search Phase of Neural Architecture Search
• A Practical Survey on Faster and Lighter Transformers
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Carbon Emissions and Large Neural Network Training
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Best Practices for Scientific Research on Neural Architecture Search
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• Very Deep Transformers for Neural Machine Translation
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Semi-Supervised Neural Architecture Search
• DeLighT: Deep and Light-weight Transformer
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Sample-Efficient Neural Architecture Search by Learning Action Space
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Taming Sparsely Activated Transformer with Stochastic Experts
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Multi-head or Single-head? An Empirical Comparison for Transformer Training
• Neural Language Generation: Formulation, Methods, and Evaluation
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild
• Automated Concatenation of Embeddings for Structured Prediction
• How to 0wn NAS in Your Spare Time
• AutoFormer: Searching Transformers for Visual Recognition
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Multi-branch Attentive Transformer
• Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
• Searching for Efficient Multi-Stage Vision Transformers
• Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Primer: Searching for Efficient Transformers for Language Modeling
• Evolving Attention with Residual Convolutions
• Neural Machine Translation: Challenges, Progress and Future
• AutoCaption: Image Captioning with Neural Architecture Search
• Balanced One-shot Neural Architecture Optimization
• Scalable Transformers for Neural Machine Translation
• Language Models are Good Translators
• The Benchmark Lottery
• Proof of Learning (PoLe): Empowering Machine Learning with Consensus Building on Blockchains
• Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search
• Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
• Evaluating Empathetic Chatbots in Customer Service Settings
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Are Labels Necessary for Neural Architecture Search?
• Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention
• Evolving to learn: discovering interpretable plasticity rules for spiking networks
• AutoRC: Improving BERT Based Relation Classification Models via Architecture Search
• NAS-Count: Counting-by-Density with Neural Architecture Search
• A Large-Scale Chinese Short-Text Conversation Dataset
• SACT: Self-Aware Multi-Space Feature Composition Transformer for Multinomial Attention for Video Captioning
• Generic Neural Architecture Search via Regression
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Multi-Pass Transformer for Machine Translation
• AutoTrans: Automating Transformer Design via Reinforced Architecture Search
• IOT: Instance-wise Layer Reordering for Transformer Structures
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• Evolving Multi-Resolution Pooling CNN for Monaural Singing Voice Separation
• Neural Machine Translation: A Review and Survey
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Selective Knowledge Distillation for Neural Machine Translation
• You May Not Need Order in Time Series Forecasting
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models
• Physics-Informed Neural State Space Models via Learning and Evolution
• Rapid Model Architecture Adaption for Meta-Learning
• Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging
• BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention
• Vocabulary Learning via Optimal Transport for Neural Machine Translation
• Towards Accurate and Compact Architectures via Neural Architecture Transformer
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• Learning Architectures from an Extended Search Space for Language Modeling
• ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
• Neural Architecture Search based on Cartesian Genetic Programming Coding Method
• Extremely Low Footprint End-to-End ASR System for Smart Device
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• Grounded Graph Decoding Improves Compositional Generalization in Question Answering
• A Survey on Green Deep Learning
• RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models
• SQALER: Scaling Question Answering by Decoupling Multi-Hop and Logical Reasoning
• Learning Algebraic Recombination for Compositional Generalization
• Residual Tree Aggregation of Layers for Neural Machine Translation
• AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
• Recurrent multiple shared layers in Depth for Neural Machine Translation
• UdS Submission for the WMT 19 Automatic Post-Editing Task
• The NiuTrans System for WNGT 2020 Efficiency Task
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
• RankNAS: Efficient Neural Architecture Search by Pairwise Ranking
• Two-Headed Monster And Crossed Co-Attention Networks