3 papers
cs.CL2025
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic
Yuval Reif, Guy Kaplan, Roy Schwartz
Large language models (LLMs) often encode word-form variation (e.g., walk vs. walked) as linear directions in the embedding space. However, standard tokenization algorithms treat s…
cs.CL2025
SpeLLM: Character-Level Multi-Head Decoding
Amit Ben-Artzy, Roy Schwartz
Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this p…
cs.CL2024
Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers
Amit Ben-Artzy, Roy Schwartz
In decoder-based LLMs, the representation of a given layer serves two purposes: as input to the next layer during the computation of the current token; and as input to the attentio…