Decoding-Free Sampling Strategies for LLM Marginalization

David Pohl, Marco Cognetta, Junyoung Lee +1

Modern language models operate on subword-tokenized text in order to make a trade-off between model size, inference speed, and vocabulary coverage. A side effect of this is that, d…