Word2Bits - Quantized Word Vectors
arXiv:1803.05651
Abstract
Word vectors require significant amounts of memory and storage, posing issues to resource limited devices like mobile phones and GPUs. We show that high quality quantized word vectors using 1-2 bits per parameter can be learned by introducing a quantization function into Word2Vec. We furthermore show that training with the quantization function acts as a regularizer. We train word vectors on English Wikipedia (2017) and evaluate them on standard word similarity and analogy tasks and on question answering (SQuAD). Our quantized word vectors not only take 8-16x less space than full precision (32 bit) word vectors but also outperform them on word similarity tasks and question answering.
References in corpus (9)
- Efficient Estimation of Word Representations in Vector Space
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Compressing Neural Networks with the Hashing Trick
- Adding Gradient Noise Improves Learning for Very Deep Networks
- Reading Wikipedia to Answer Open-Domain Questions
- Estimating or Propagating Gradients Through Stochastic Neurons
- High-Accuracy Low-Precision Training
- Compressing Word Embeddings via Deep Compositional Code Learning
Cited by in corpus (7)
- Tensorized Embedding Layers for Efficient Model Compression
- GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
- Binarized Canonical Polyadic Decomposition for Knowledge Graph Completion
- Binarized Knowledge Graph Embeddings
- Quantized Neural Network Inference with Precision Batching
- A Survey on Green Deep Learning
- Hamming Sentence Embeddings for Information Retrieval