Neural Architecture Search for Sentence Classification with BERT

Abstract

Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.

References in corpus (10)

• Practical Bayesian Optimization of Machine Learning Algorithms
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• AutoML: A Survey of the State-of-the-Art
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
• Parameter-Efficient Transfer Learning for NLP
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Model-based Asynchronous Hyperparameter and Neural Architecture Search