9 papers
From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models
Masanari Oi, Koki Maeda, Ryuto Koike +3
While multimodal large language models (MLLMs) have made substantial progress in single-image spatial reasoning, multi-image spatial reasoning, which requires integration of inform…
Decoding-Free Sampling Strategies for LLM Marginalization
David Pohl, Marco Cognetta, Junyoung Lee +1
Modern language models operate on subword-tokenized text in order to make a trade-off between model size, inference speed, and vocabulary coverage. A side effect of this is that, d…
WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models
Issa Sugiura, Shuhei Kurita, Yusuke Oda +3
Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing English-only caption filters and pretraini…
Bit-level BPE: Below the byte boundary
Sangwhan Moon, Tatsuya Hiraoka, Naoaki Okazaki
Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragma…
Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model
Keito Sasagawa, Koki Maeda, Issa Sugiura +3
To develop high-performing Visual Language Models (VLMs), it is essential to prepare multimodal resources, such as image-text pairs, interleaved data, and instruction data. While m…
Tokenization as Finite-State Transduction
Marco Cognetta, Naoaki Okazaki
Tokenization is the first step in modern neural language model pipelines where an input text is converted to a sequence of subword tokens. We introduce from first principles a fini…
Distributional Properties of Subword Regularization
Marco Cognetta, Vilém Zouhar, Naoaki Okazaki
Subword regularization, used widely in NLP, improves model performance by reducing the dependency on exact tokenizations, augmenting the training corpus, and exposing the model to…
Two Counterexamples to Tokenization and the Noiseless Channel
Marco Cognetta, Vilém Zouhar, Sangwhan Moon +1
In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer w…
Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
Koki Maeda, Shuhei Kurita, Taiki Miyanishi +1
Given the accelerating progress of vision and language modeling, accurate evaluation of machine-generated image captions remains critical. In order to evaluate captions more closel…