1 citations · 1 across the 5 of their papers we have counts for
6 papers
LightMTP: Lightweight Latent Multi-Token Prediction
Tamara Czinczoll, Julie Kallini, Gerard de Melo +1
Next-token prediction (NTP) is the standard pretraining objective for large language models, yet it provides an explicit training signal only for the immediate next token, which ca…
Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet
Milan Miletić, Julie Kallini, Ekaterina Shutova
Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches f…
Fast Byte Latent Transformer
Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5
Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte auto…
False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
Julie Kallini, Dan Jurafsky, Christopher Potts +1
Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages. Does token overlap facilitate cross-lingual transfer or instead introduce…
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
Julie Kallini, Shikhar Murty, Christopher D. Manning +2
Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across dif…
Mission: Impossible Language Models
Julie Kallini, Isabel Papadimitriou, Richard Futrell +2
Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. Howe…