Memory-Efficient Differentiable Transformer Architecture Search

Abstract

Differentiable architecture search (DARTS) is successfully applied in many vision tasks. However, directly using DARTS for Transformers is memory-intensive, which renders the search process infeasible. To this end, we propose a multi-split reversible network and combine it with DARTS. Specifically, we devise a backpropagation-with-reconstruction algorithm so that we only need to store the last layer's outputs. By relieving the memory burden for DARTS, it allows us to search with larger hidden size and more candidate operations. We evaluate the searched architecture on three sequence-to-sequence datasets, i.e., WMT'14 English-German, WMT'14 English-French, and WMT'14 English-Czech. Experimental results show that our network consistently outperforms standard Transformers across the tasks. Moreover, our method compares favorably with big-size Evolved Transformers, reducing search computation by an order of magnitude.

Accepted by Findings of ACL 2021

References in corpus (15)

• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• SNAS: Stochastic Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Hierarchical Representations for Efficient Architecture Search
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Neural Architecture Transfer
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Understanding Back-Translation at Scale
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Multiscale Deep Equilibrium Models
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• DSNAS: Direct Neural Architecture Search without Parameter Retraining