Showing cs.CLShow all
3 papers · 1 filter
cs.CL2025
Decoding-Free Sampling Strategies for LLM Marginalization
David Pohl, Marco Cognetta, Junyoung Lee +1
Modern language models operate on subword-tokenized text in order to make a trade-off between model size, inference speed, and vocabulary coverage. A side effect of this is that, d…
cs.CL2024
Tokenization as Finite-State Transduction
Marco Cognetta, Naoaki Okazaki
Tokenization is the first step in modern neural language model pipelines where an input text is converted to a sequence of subword tokens. We introduce from first principles a fini…
cs.CL2024
Distributional Properties of Subword Regularization
Marco Cognetta, Vilém Zouhar, Naoaki Okazaki
Subword regularization, used widely in NLP, improves model performance by reducing the dependency on exact tokenizations, augmenting the training corpus, and exposing the model to…