Publications (25)
llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length
Issa Sugiura, Kouta Nakayama, Yusuke Oda
Encoder-only transformer models like BERT are widely adopted as a pre-trained backbone for tasks like sentence classification and retrieval. However, pretraining of encoder models…
WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models
Issa Sugiura, Shuhei Kurita, Yusuke Oda +3
Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing English-only caption filters and pretraini…
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
Xuyang Cao, Qianying Liu, Chuan Xiao +7
In multilingual pretraining, the test loss of a pretrained model is heavily influenced by the proportion of each language in the pretraining data, namely the \textit{language mixtu…
Instability in Downstream Task Performance During LLM Pretraining
Yuto Nishida, Masaru Isonuma, Yusuke Oda
When training large language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest va…
Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification
Koichi Akabe, Shunsuke Kanda, Yusuke Oda +1
This paper proposes an approach to improve the runtime efficiency of Japanese tokenization based on the pointwise linear classification (PLC) framework, which formulates the whole…
Refactoring Programs Using Large Language Models with Few-Shot Examples
Atsushi Shirafuji, Yusuke Oda, Jun Suzuki +2
A less complex and more straightforward program is a crucial factor that enhances its maintainability and makes writing secure and bug-free programs easier. However, due to its hea…
DyNet: The Dynamic Neural Network Toolkit
Graham Neubig, Chris Dyer, Yoav Goldberg +22
We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits l…
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
Daichi Yashima, Shuhei Kurita, Yusuke Oda +1
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge. In this st…
ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
Daichi Yashima, Shuhei Kurita, Yusuke Oda +3
In this study, we focus on video captioning by fully open multimodal large language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate…
Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
Issa Sugiura, Shuhei Kurita, Yusuke Oda +1
Speech Language Models (SpeechLMs) model tokenized speech to capture both semantic and acoustic information. When neural audio codecs based on Residual Vector Quantization (RVQ) ar…
Findings of the Second Workshop on Neural Machine Translation and Generation
Alexandra Birch, Andrew Finch, Minh-Thang Luong +2
This document describes the findings of the Second Workshop on Neural Machine Translation and Generation, held in concert with the annual conference of the Association for Computat…
An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation
Makoto Morishita, Yusuke Oda, Graham Neubig +3
Training of neural machine translation (NMT) models usually uses mini-batches for efficiency purposes. During the mini-batched training process, it is necessary to pad shorter sent…
HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers
Issa Sugiura, Shuhei Kurita, Yusuke Oda +1
Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, n…
JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Issa Sugiura, Koki Maeda, Shuhei Kurita +3
Reliable evaluation is essential for the development of vision-language models (VLMs). However, Japanese VQA benchmarks have undergone far less iterative refinement than their Engl…
Are Prompt-based Models Clueless?
Pride Kavumba, Ryo Takahashi, Yusuke Oda
Finetuning large pre-trained language models with a task-specific head has advanced the state-of-the-art on many natural language understanding benchmarks. However, models with a t…
Exploring the Robustness of Large Language Models for Solving Programming Problems
Atsushi Shirafuji, Yutaka Watanobe, Takumi Ito +4
Using large language models (LLMs) for source code has recently gained attention. LLMs, such as Transformer-based models like Codex and ChatGPT, have been shown to be highly capabl…
Findings of the Third Workshop on Neural Generation and Translation
Hiroaki Hayashi, Yusuke Oda, Alexandra Birch +5
This document describes the findings of the Third Workshop on Neural Generation and Translation, held in concert with the annual conference of the Empirical Methods in Natural Lang…
Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
Yuto Harada, Yusuke Yamauchi, Yusuke Oda +3
Supervised fine-tuning (SFT) is a critical step in aligning large language models (LLMs) with human instructions and values, yet many aspects of SFT remain poorly understood. We tr…
Neural Machine Translation via Binary Code Prediction
Yusuke Oda, Philip Arthur, Graham Neubig +2
In this paper, we propose a new method for calculating the output layer in neural machine translation systems. The method is based on predicting a binary code for each word and can…
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
LLM-jp, :, Akiko Aizawa +80
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and stron…
Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models
Issa Sugiura, Keito Sasagawa, Keisuke Nakao +8
Developing vision-language models (VLMs) that generalize across diverse tasks requires large-scale training datasets with diverse content. In English, such datasets are typically c…
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
Taishi Nakamura, Takuya Akiba, Kazuki Fujii +3
The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that init…
Engineering faster double-array Aho-Corasick automata
Shunsuke Kanda, Koichi Akabe, Yusuke Oda
Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This paper studies efficient implementati…
Detecting Sensitive Personal Information in Japanese Pre-Training Corpora for Large Language Models
Rei Minamoto, Yusuke Oda, Daisuke Kawahara
Sensitive personal information can appear in large-scale pre-training corpora for large language models (LLMs). Detecting and filtering such information is therefore essential to e…
HiFlow: Tokenization-Free Scale-Wise Autoregressive Policy Learning via Flow Matching
Daichi Yashima, Koki Seno, Shuhei Kurita +2
Coarse-to-fine autoregressive modeling has recently shown strong promise for visuomotor policy learning, combining the inference efficiency of autoregressive methods with the globa…