Self-organized Hierarchical Softmax

Abstract

We propose a new self-organizing hierarchical softmax formulation for neural-network-based language models over large vocabularies. Instead of using a predefined hierarchical structure, our approach is capable of learning word clusters with clear syntactical and semantic meaning during the language model training process. We provide experiments on standard benchmarks for language modeling and sentence compression tasks. We find that this approach is as fast as other efficient softmax approximations, while achieving comparable or even better performance relative to similar full softmax models.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Exploring the Limits of Language Modeling
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Learning Longer Memory in Recurrent Neural Networks
• Classes for Fast Maximum Entropy Training
• Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Neural Headline Generation with Sentence-wise Optimization

Cited by in corpus (3)

• Prototype Memory for Large-scale Face Representation Learning
• Multi-modal Geolocation Estimation Using Deep Neural Networks
• Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation