11 papers
CMDR: Contextual Multimodal Document Retrieval
Ryota Tanaka, Taku Hasegawa, Kyosuke Nishida
Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily ev…
Debiasing Reward Models via Causally Motivated Inference-Time Intervention
Kazutoshi Shinoda, Kosuke Nishida, Kyosuke Nishida
Reward models (RMs) play a central role in aligning large language models (LLMs) with human preferences. However, RMs are often sensitive to spurious features such as response leng…
Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
Haruka Kawasaki, Ryota Tanaka, Kyosuke Nishida
Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning…
Lossless Vocabulary Reduction for Auto-Regressive Language Models
Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida +4
Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularl…
Let's Put Ourselves in Sally's Shoes: Shoes-of-Others Prefilling Improves Theory of Mind in Large Language Models
Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida +4
Recent studies have shown that Theory of Mind (ToM) in large language models (LLMs) has not reached human-level performance yet. Since fine-tuning LLMs on ToM datasets often degrad…
Can LLMs Detect Their Own Hallucinations?
Sora Kadotani, Kosuke Nishida, Kyosuke Nishida
Large language models (LLMs) can generate fluent responses, but sometimes hallucinate facts. In this paper, we investigate whether LLMs can detect their own hallucinations. We form…