7 citations · 20 across the 16 of their papers we have counts for
17 papers
LLM-jp-4 Technical Report
LLM-jp, :, Chaoran Liu +16
This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-…
Evaluating Multiple LLM Generations with Validated Task Coverage
Florian Le Bronnec, Rio Yokota
Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation settings, however, still focus…
LLMs as Acquisition Policies for Finite-Pool Materials Optimization: A Controlled Study
Dino-Rober Demir, Florian Le Bronnec, Rio Yokota
Discovering materials with desirable properties often requires searching large candidate spaces while experimental or computational evaluations remain costly. Active learning addre…
Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation
Le Bronnec Florian, Alexandre Verine, Rio Yokota +1
LLMs for code generation are commonly evaluated in repeated-sampling settings using Pass@k, where multiple candidate programs are executed against unit tests under a finite samplin…
Evolutionary Context Search for Automated Skill Acquisition
Qi Sun, Stefan Nielsen, Rio Yokota +1
Large Language Models cannot reliably acquire new knowledge post-deployment -- even when relevant text resources exist, models fail to transform them into actionable knowledge with…
On the Optimal Reasoning Length for RL-Trained Language Models
Daisuke Nohara, Taishi Nakamura, Rio Yokota
Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although l…
Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura +4
Empirical scaling laws have driven the evolution of large language models (LLMs), yet their coefficients shift whenever the model architecture or data pipeline changes. Mixture-of-…
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Kazuki Fujii, Yukito Tajima, Sakae Mizuki +14
The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce…
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
Taishi Nakamura, Takuya Akiba, Kazuki Fujii +3
The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that init…
Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
Youmi Ma, Sakae Mizuki, Kazuki Fujii +12
Instruction tuning is crucial for enabling Large Language Models (LLMs) to solve real-world tasks. Prior work has shown the effectiveness of instruction-tuning data synthesized sol…
Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs
Koshiro Saito, Sakae Mizuki, Masanari Ohi +11
Why do we build local large language models (LLMs)? What should a local LLM learn from the target language? Which abilities can be transferred from other languages? Do language-spe…
Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
Kazuki Fujii, Taishi Nakamura, Rio Yokota
Large Language Models (LLMs) have attracted significant attention due to their human-like language understanding and generation capabilities, as well as their applicability across…
Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
Kazuki Fujii, Kohei Watanabe, Rio Yokota
In large language model (LLM) training, several parallelization strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), as well as Sequence…
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
LLM-jp, :, Akiko Aizawa +80
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and stron…
Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
Kazuki Fujii, Taishi Nakamura, Mengsay Loem +7
Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and red…
Building a Large Japanese Web Corpus for Large Language Models
Naoaki Okazaki, Kakeru Hattori, Hirai Shota +7
Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the qua…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…