6 papers
Timesteps of Mamba Align with Human Reading Times
Yuji Yamamoto, Shinnosuke Isono, Yoshinobu Kawahara +1
This study demonstrates an alignment of per-word processing time in a popular state-space language model Mamba and human readers. In Mamba, the recurrent state transition at each l…
Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings
Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi +2
For constructing text embeddings, mean pooling, which averages token embeddings, is the standard approach. This paper examines whether mean pooling actually works well in real mode…
Linear Representations of Hierarchical Concepts in Language Models
Masaki Sakata, Benjamin Heinzerling, Takumi Ito +2
We investigate how and to what extent hierarchical relations (e.g., Japan Eastern Asia Asia) are encoded in the internal representations of language models. Bui…
On Entity Identification in Language Models
Masaki Sakata, Benjamin Heinzerling, Sho Yokoi +2
We analyze the extent to which internal representations of language models (LMs) identify and distinguish mentions of named entities, focusing on the many-to-many correspondence be…
Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi +1
Transformers are ubiquitous in wide tasks. Interpreting their internals is a pivotal goal. Nevertheless, their particular components, feed-forward (FF) blocks, have typically been…
Subspace Representations for Soft Set Operations and Sentence Similarities
Yoichi Ishibashi, Sho Yokoi, Katsuhito Sudoh +1
In the field of natural language processing (NLP), continuous vector representations are crucial for capturing the semantic meanings of individual words. Yet, when it comes to the…