5 papers
LangMAP: A Language-Adaptive Approach to Tokenization
Clara Meister, Suchir Salhan, Andrzej Szablewski +3
Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new…
Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs
Charles Westphal, Timothy Douglas, Keivan Navaie +2
Large language models can be fine-tuned to encode prompt-borne secrets into fluent, seemingly benign outputs. This creates a steganographic exfiltration risk that is difficult to d…
Tokenisation via Convex Relaxations
Jan Tempus, Philip Whittington, Craig W. Schmidt +2
Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions w…
Tokenisation over Bounded Alphabets is Hard
Violeta Kastreva, Philip Whittington, Dennis Komm +1
Recent works have shown that tokenisation is NP-complete. However, these works assume tokenisation is applied to inputs with unboundedly large alphabets -- an unrealistic assumptio…
Tokenisation is NP-Complete
Philip Whittington, Gregor Bachmann, Tiago Pimentel
In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most symbols by either finding a vocabulary d…