Softmax Reparameterization for Output-Head Quantization

Asim Kadav, Christian Flores, Chirag Arora +5

Large vocabularies make output heads a substantial inference cost in small language models. We introduce softmax reparameterization, a post-training method that searches over funct…