1 citations · 2 across the 12 of their papers we have counts for
12 papers
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Raito Kiya, Satoki Ohashi, Kosuke Sato +6
Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently…
Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
Mutsumi Sasaki, Go kamoda, Ryosuke Takahashi +4
Quantities with measurement units, such as 110 cm and 1.2 m, require language models (LMs) to combine a numeral with a symbolic unit scale. Here, we study how LMs compare such quan…
Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models
Haruto Yoshida, Keito Kudo, Yoichi Aoki +4
Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements,…
Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki
Mutsumi Sasaki, Go Kamoda, Ryosuke Takahashi +4
Temporal reasoning and knowledge are essential capabilities for language models (LMs). While much prior work has analyzed and improved temporal reasoning in LMs, most studies have…
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference
Go Kamoda, Benjamin Heinzerling, Tatsuro Inaba +3
According to the stages-of-inference hypothesis, early layers of language models map their subword-tokenized input, which does not necessarily correspond to a linguistically meanin…
FinchGPT: a Transformer based language model for birdsong analysis
Kosei Kobayashi, Kosuke Matsuzaki, Masaya Taniguchi +3
The long-range dependencies among the tokens, which originate from hierarchical structures, are a defining hallmark of human language. However, whether similar dependencies exist w…
Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection
Subaru Kimura, Ryota Tanaka, Shumpei Miyawaki +2
We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose…
The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models
Ryosuke Takahashi, Go Kamoda, Benjamin Heinzerling +2
Language models (LMs) encode world knowledge in their internal parameters through training. However, LMs may learn personal and confidential information from the training data, lea…
First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning
Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi +4
Multi-step reasoning instruction, such as chain-of-thought prompting, is widely adopted to explore better language models (LMs) performance. We report on the systematic strategy th…
ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation
Ana Brassard, Benjamin Heinzerling, Keito Kudo +2
Evaluating the quality of free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their…
J-UniMorph: Japanese Morphological Annotation through the Universal Feature Schema
Kosuke Matsuzaki, Masaya Taniguchi, Kentaro Inui +1
We introduce a Japanese Morphology dataset, J-UniMorph, developed based on the UniMorph feature schema. This dataset addresses the unique and rich verb forms characteristic of the…
Test-time Augmentation for Factual Probing
Go Kamoda, Benjamin Heinzerling, Keisuke Sakaguchi +1
Factual probing is a method that uses prompts to test if a language model "knows" certain world knowledge facts. A problem in factual probing is that small changes to the prompt ca…