6 papers
Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge
Ying Zhang, Benjamin Heinzerling, Dongyuan Li +1
While fine-tuning is the standard for injecting factual knowledge into large language models (LLMs), the mechanisms enabling reliable fact recall via unseen queries remain poorly u…
How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders
Tatsuro Inaba, Go Kamoda, Kentaro Inui +5
This study explores how bilingual language models develop complex internal representations. We employ sparse autoencoders to analyze internal representations of bilingual language…
Do LLMs Need to Think in One Language? Correlation between Latent Language and Task Performance
Shintaro Ozaki, Tatsuya Hiraoka, Hiroto Otake +8
Large Language Models (LLMs) are known to process information using a proficient internal language consistently, referred to as latent language, which may differ from the input or…
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference
Go Kamoda, Benjamin Heinzerling, Tatsuro Inaba +3
According to the stages-of-inference hypothesis, early layers of language models map their subword-tokenized input, which does not necessarily correspond to a linguistically meanin…
The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces
Ahmed Oumar El-Shangiti, Tatsuya Hiraoka, Hilal AlQuabeh +2
This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions invo…
RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles
Munachiso Nwadike, Zangir Iklassov, Toluwani Aremu +6
We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirection…