6 papers · 1 filter
Revisiting Non-Adaptive Set Membership Schemes: Removing the Logarithmic Factor
Mirza Galib Anwarul Husain Baig, Aman Kumar, Pawan Kumar Saw +1
We study non-adaptive bit-probe schemes for static set membership. Each query reads a fixed number of bits and answers by strict majority. We revisit the random bipartite-graph con…
More Than Words: Compositional Tokenization for Efficient Language Models
Yuval Reif, Guy Kaplan, Roy Schwartz
Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short E…
Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
Amit Ben-Artzy, Roy Schwartz
Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers despite relying on fundamentally different architectures. This raise…
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
Federico Marcuzzi, Xuefei Ning, Roy Schwartz +1
As Large Language Models are increasingly deployed in critical applications, robustly evaluating their social biases is paramount. However, the current literature suffers from wide…
Post-training is (Massive) Supervised Learning
Michael Hassid, Yossi Adi, Roy Schwartz
The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL. In this position paper, we argue that this methodology effe…
Why Fine-Tuning Encourages Hallucinations and How to Fix It
Guy Kaplan, Zorik Gekhman, Zhen Zhu +5
Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning…