Showing cs.CLShow all
2 papers · 1 filter
cs.CL2025
Toward a Theory of Tokenization in LLMs
Nived Rajaraman, Jiantao Jiao, Kannan Ramchandran
While there has been a large body of research attempting to circumvent tokenization for language modeling (Clark et al., 2022; Xue et al., 2022), the current consensus is that it i…
cs.CL2024
EmbedLLM: Learning Compact Representations of Large Language Models
Richard Zhuang, Tianhao Wu, Zhaojin Wen +3
With hundreds of thousands of language models available on Huggingface today, efficiently evaluating and utilizing these models across various downstream, tasks has become increasi…