12 papers
The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs
Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh +2
Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting. We investigate whether this reflects missing internal…
Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs
Kareem Elozeiri, Mervat Abassy, Omar Kallas +4
Dialectal data are scarce relative to Modern Standard Arabic (MSA), causing Arabic LLMs to overproduce MSA and struggle with dialectally accurate generation. This raises a fundamen…
Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings
Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi +2
For constructing text embeddings, mean pooling, which averages token embeddings, is the standard approach. This paper examines whether mean pooling actually works well in real mode…
A Robust Evaluation of Probe Robustness: Lessons for Reliable OOD Uncertainty Quantification
Joe Stacey, Hadas Orgad, Kentaro Inui +2
Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation, motivating a growing interest in efficient probe-based appr…
WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention
Ruben Solozabal, Velibor Bojkovic, Hilal Alquabeh +3
State-space models (SSMs) have emerged as a powerful foundation for long-range sequence modeling, with the HiPPO framework showing that continuous-time projection operators can be…
Sycophancy Hides Linearly in the Attention Heads
Rifo Genadi, Munachiso Nwadike, Nurdaulet Mukhituly +3
We find that correct-to-incorrect sycophancy signals are most linearly separable within multi-head attention activations. Motivated by the linear representation hypothesis, we trai…
Uncovering the Spectral Bias in Diagonal State Space Models
Ruben Solozabal, Velibor Bojkovic, Hilal AlQuabeh +2
Current methods for initializing state space models (SSMs) parameters mainly rely on the \textit{HiPPO framework}, which is based on an online approximation of orthogonal polynomia…
Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
Nhi Hoai Doan, Tatsuya Hiraoka, Kentaro Inui
This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In…
Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
Muhammad Cendekia Airlangga, Hilal AlQuabeh, Munachiso S Nwadike +1
We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time. Applying struct…
Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters
Tatsuya Hiraoka, Kentaro Inui
Large language models (LLMs) can spell out tokens character by character with high accuracy, yet they struggle with more complex character-level tasks, such as identifying composit…
Mechanistic Insights into Grokking from the Embedding Layer
H. V. AlquBoj, Hilal AlQuabeh, Velibor Bojkovic +2
Grokking, a delayed generalization in neural networks after perfect training performance, has been observed in Transformers and MLPs, but the components driving it remain underexpl…
Number Representations in LLMs: A Computational Parallel to Human Perception
H. V. AlquBoj, Hilal AlQuabeh, Velibor Bojkovic +4
Humans are believed to perceive numbers on a logarithmic mental number line, where smaller values are represented with greater resolution than larger ones. This cognitive bias, sup…