2 papers
cs.CL2025
Decoding-Free Sampling Strategies for LLM Marginalization
David Pohl, Marco Cognetta, Junyoung Lee +1
Modern language models operate on subword-tokenized text in order to make a trade-off between model size, inference speed, and vocabulary coverage. A side effect of this is that, d…
cs.CL2024
Tokenization as Finite-State Transduction
Marco Cognetta, Naoaki Okazaki
Tokenization is the first step in modern neural language model pipelines where an input text is converted to a sequence of subword tokens. We introduce from first principles a fini…