Relaxed Softmax for learning from Positive and Unlabeled data
arXiv:1909.08079 · doi:10.1145/3298689.3347034
Abstract
In recent years, the softmax model and its fast approximations have become the de-facto loss functions for deep neural networks when dealing with multi-class prediction. This loss has been extended to language modeling and recommendation, two fields that fall into the framework of learning from Positive and Unlabeled data. In this paper, we stress the different drawbacks of the current family of softmax losses and sampling schemes when applied in a Positive and Unlabeled learning setup. We propose both a Relaxed Softmax loss (RS) and a new negative sampling scheme based on Boltzmann formulation. We show that the new training objective is better suited for the tasks of density estimation, item similarity and next-event prediction by driving uplifts in performance on textual and recommendation datasets against classical softmax.
arXiv comment9 pages, 5 figures, 2 tables, published at RecSys 2019
References in corpus (13)
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- Distributed Representations of Sentences and Documents
- Enriching Word Vectors with Subword Information
- A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
- Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation
- A Benchmark and Comparison of Active Learning for Logistic Regression
- subgraph2vec: Learning Distributed Representations of Rooted Sub-graphs from Large Graphs
- Learning Longer-term Dependencies in RNNs with Auxiliary Losses
- On Using Very Large Target Vocabulary for Neural Machine Translation
- Swivel: Improving Embeddings by Noticing What's Missing
- Learning Algorithms for Active Learning
- Efficient and Parsimonious Agnostic Active Learning