PAC-Bayes Analysis of Sentence Representation
arXiv:1902.04247
Abstract
Learning sentence vectors from an unlabeled corpus has attracted attention because such vectors can represent sentences in a lower dimensional and continuous space. Simple heuristics using pre-trained word vectors are widely applied to machine learning tasks. However, they are not well understood from a theoretical perspective. We analyze learning sentence vectors from a transfer learning perspective by using a PAC-Bayes bound that enables us to understand existing heuristics. We show that simple heuristics such as averaging and inverse document frequency weighted averaging are derived by our formulation. Moreover, we propose novel sentence vector learning algorithms on the basis of our PAC-Bayes analysis.
arXiv commentfix styles
References in corpus (7)
- Distributed Representations of Words and Phrases and their Compositionality
- Auto-Encoding Variational Bayes
- Natural Language Processing (almost) from Scratch
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- A PAC-Bayesian bound for Lifelong Learning
- Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory