TextNAS: A Neural Architecture Search Space tailored for Text Representation
arXiv:1912.10729
Abstract
Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.
References in corpus (15)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Neural Architecture Search with Reinforcement Learning
- Character-level Convolutional Networks for Text Classification
- A C-LSTM Neural Network for Text Classification
- Efficient Neural Architecture Search via Parameter Sharing
- A large annotated corpus for learning natural language inference
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Random Search and Reproducibility for Neural Architecture Search
- Recurrent Convolutional Neural Networks for Discourse Compositionality
- Progressive Neural Architecture Search
- Distance-based Self-Attention Network for Natural Language Inference
- Enhancing Sentence Embedding with Generalized Pooling
- A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
- Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding
- Shortcut-Stacked Sentence Encoders for Multi-Domain Inference