6 papers · 1 filter
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
Federico Marcuzzi, Xuefei Ning, Roy Schwartz +1
As Large Language Models are increasingly deployed in critical applications, robustly evaluating their social biases is paramount. However, the current literature suffers from wide…
Post-training is (Massive) Supervised Learning
Michael Hassid, Yossi Adi, Roy Schwartz
The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL. In this position paper, we argue that this methodology effe…
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic
Yuval Reif, Guy Kaplan, Roy Schwartz
Large language models (LLMs) often encode word-form variation (e.g., walk vs. walked) as linear directions in the embedding space. However, standard tokenization algorithms treat s…
Why Fine-Tuning Encourages Hallucinations and How to Fix It
Guy Kaplan, Zorik Gekhman, Zhen Zhu +5
Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning…
SpeLLM: Character-Level Multi-Head Decoding
Amit Ben-Artzy, Roy Schwartz
Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this p…
Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers
Amit Ben-Artzy, Roy Schwartz
In decoder-based LLMs, the representation of a given layer serves two purposes: as input to the next layer during the computation of the current token; and as input to the attentio…