A Comparative Study on Regularization Strategies for Embedding-based Neural Networks
arXiv:1508.03721
Abstract
This paper aims to compare different regularization strategies to address a common phenomenon, severe overfitting, in embedding-based neural networks for NLP. We chose two widely studied neural models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout. We also emphasized on incremental hyperparameter tuning, and combining different regularizations. The results provide a picture on tuning hyperparameters for neural NLP models.
arXiv commentEMNLP '15
References in corpus (4)
Cited by in corpus (5)
- How Transferable are Neural Networks in NLP Applications?
- Natural Language Inference by Tree-Based Convolution and Heuristic Matching
- NISER: Normalized Item and Session Representations to Handle Popularity Bias
- Backward and Forward Language Modeling for Constrained Sentence Generation
- Text classification with word embedding regularization and soft similarity measure