On the Effective Use of Pretraining for Natural Language Inference
arXiv:1710.02076
Abstract
Neural networks have excelled at many NLP tasks, but there remain open questions about the performance of pretrained distributed word representations and their interaction with weight initialization and other hyperparameters. We address these questions empirically using attention-based sequence-to-sequence models for natural language inference (NLI). Specifically, we compare three types of embeddings: random, pretrained (GloVe, word2vec), and retrofitted (pretrained plus WordNet information). We show that pretrained embeddings outperform both random and retrofitted ones in a large NLI corpus. Further experiments on more controlled data sets shed light on the contexts for which retrofitted embeddings can be useful. We also explore two principled approaches to initializing the rest of the model parameters, Gaussian and orthogonal, showing that the latter yields gains of up to 2.9% in the NLI task.
arXiv commentThis manuscript dates from late Winter 2016
References in corpus (7)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
- Reasoning about Entailment with Neural Attention
- A large annotated corpus for learning natural language inference
- Long Short-Term Memory-Networks for Machine Reading
- Learning Natural Language Inference with LSTM