Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

Abstract

Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance. However, little is published which parameters and design choices should be evaluated or selected making the correct hyperparameter optimization often a "black art that requires expert experiences" (Snoek et al., 2012). In this paper, we evaluate the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks (POS, Chunking, NER, Entity Recognition, and Event Detection). We evaluated over 50.000 different setups and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or the number of recurrent units, are of minor importance. We give a recommendation on a configuration that performs well among different tasks.

34 pages. 9 page version of this paper published at EMNLP 2017

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• Natural Language Processing (almost) from Scratch
• Enriching Word Vectors with Subword Information
• Practical recommendations for gradient-based training of deep architectures
• TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

Cited by in corpus (47)

• An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks
• Hierarchically-Refined Label Attention Network for Sequence Labeling
• Greenhouse Gas Emission Prediction on Road Network using Deep Sequence Learning
• Few-shot Learning for Named Entity Recognition in Medical Text
• Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work
• SciBERT: A Pretrained Language Model for Scientific Text
• Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches
• Short-term daily precipitation forecasting with seasonally-integrated autoencoder
• A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
• Multilingual Training and Cross-lingual Adaptation on CTC-based Acoustic Model
• Improving Named Entity Recognition by Jointly Learning to Disambiguate Morphological Tags
• De-identifying Australian Hospital Discharge Summaries: An End-to-End Framework using Ensemble of Deep Learning Models
• Neural Modeling for Named Entities and Morphology (NEMO^2)
• Predicting Future Sales of Retail Products using Machine Learning
• Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher
• Prior Knowledge Driven Label Embedding for Slot Filling in Natural Language Understanding
• AliCoCo: Alibaba E-commerce Cognitive Concept Net
• Learning representations for multivariate time series with missing data using Temporal Kernelized Autoencoders
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study
• Comparing Rule-based, Feature-based and Deep Neural Methods for De-identification of Dutch Medical Records
• On Mini-Batch Training with Varying Length Time Series
• Approximate Cross-Validation for Structured Models
• Partially-Typed NER Datasets Integration: Connecting Practice to Theory
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling
• Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
• Raw-to-End Name Entity Recognition in Social Media
• CalibreNet: Calibration Networks for Multilingual Sequence Labeling
• Global Attention for Name Tagging
• Linguistically-Enriched and Context-Aware Zero-shot Slot Filling
• Speaker-change Aware CRF for Dialogue Act Classification
• Deep Cascade Multi-task Learning for Slot Filling in Online Shopping Assistant
• Toward Open-domain Slot Filling via Self-supervised Co-training
• Benchmarking Modern Named Entity Recognition Techniques for Free-text Health Record De-identification
• Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis
• LaTeX-Numeric: Language-agnostic Text attribute eXtraction for E-commerce Numeric Attributes
• NorNE: Annotating Named Entities for Norwegian
• Joint Learning of Word and Label Embeddings for Sequence Labelling in Spoken Language Understanding
• A Bayesian Approach for Sequence Tagging with Crowds
• Syllable-based Neural Named Entity Recognition for Myanmar Language
• Multi-Task Learning for Argumentation Mining
• ICU Mortality Prediction Using Long Short-Term Memory Networks
• A Comparative Study of Sequence Classification Models for Privacy Policy Coverage Analysis
• Comparing CNN and LSTM character-level embeddings in BiLSTM-CRF models for chemical and disease named entity recognition
• CitationIE: Leveraging the Citation Graph for Scientific Information Extraction
• DeepVar: An End-to-End Deep Learning Approach for Genomic Variant Recognition in Biomedical Literature
• Teaching a Machine to Diagnose a Heart Disease; Beginning from digitizing scanned ECGs to detecting the Brugada Syndrome (BrS)
• Quantization Loss Re-Learning Method