3 citations · 4 across the 6 of their papers we have counts for
1 paper · 1 filter
Mete Erdogan, Abhiram Gorle, Shubham Chandak +2
Large language model (LLM) tokenizers act as structured compressors: by mapping text to discrete token sequences, they determine token count (and thus compute and context usage) an…