9 papers
Revisiting Non-Adaptive Set Membership Schemes: Removing the Logarithmic Factor
Mirza Galib Anwarul Husain Baig, Aman Kumar, Pawan Kumar Saw +1
We study non-adaptive bit-probe schemes for static set membership. Each query reads a fixed number of bits and answers by strict majority. We revisit the random bipartite-graph con…
More Than Words: Compositional Tokenization for Efficient Language Models
Yuval Reif, Guy Kaplan, Roy Schwartz
Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short E…
Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
Amit Ben-Artzy, Roy Schwartz
Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers despite relying on fundamentally different architectures. This raise…
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
Federico Marcuzzi, Xuefei Ning, Roy Schwartz +1
As Large Language Models are increasingly deployed in critical applications, robustly evaluating their social biases is paramount. However, the current literature suffers from wide…
Post-training is (Massive) Supervised Learning
Michael Hassid, Yossi Adi, Roy Schwartz
The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL. In this position paper, we argue that this methodology effe…
Why Fine-Tuning Encourages Hallucinations and How to Fix It
Guy Kaplan, Zorik Gekhman, Zhen Zhu +5
Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning…
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic
Yuval Reif, Guy Kaplan, Roy Schwartz
Large language models (LLMs) often encode word-form variation (e.g., walk vs. walked) as linear directions in the embedding space. However, standard tokenization algorithms treat s…
SpeLLM: Character-Level Multi-Head Decoding
Amit Ben-Artzy, Roy Schwartz
Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this p…
Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers
Amit Ben-Artzy, Roy Schwartz
In decoder-based LLMs, the representation of a given layer serves two purposes: as input to the next layer during the computation of the current token; and as input to the attentio…