7 papers
Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
Mutsumi Sasaki, Go kamoda, Ryosuke Takahashi +4
Quantities with measurement units, such as 110 cm and 1.2 m, require language models (LMs) to combine a numeral with a symbolic unit scale. Here, we study how LMs compare such quan…
Cell-Based Representation of Relational Binding in Language Models
Qin Dai, Benjamin Heinzerling, Kentaro Inui
Understanding a discourse requires tracking entities and the relations that hold between them. While Large Language Models (LLMs) perform well on relational reasoning, the mechanis…
Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation
Joe Stacey, Hadas Orgad, Kentaro Inui +2
Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation and hallucination detection, motivating a growing interest i…
Linear Representations of Hierarchical Concepts in Language Models
Masaki Sakata, Benjamin Heinzerling, Takumi Ito +2
We investigate how and to what extent hierarchical relations (e.g., Japan Eastern Asia Asia) are encoded in the internal representations of language models. Bui…
Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki
Mutsumi Sasaki, Go Kamoda, Ryosuke Takahashi +4
Temporal reasoning and knowledge are essential capabilities for language models (LMs). While much prior work has analyzed and improved temporal reasoning in LMs, most studies have…
On Entity Identification in Language Models
Masaki Sakata, Benjamin Heinzerling, Sho Yokoi +2
We analyze the extent to which internal representations of language models (LMs) identify and distinguish mentions of named entities, focusing on the many-to-many correspondence be…