7 citations · 20 across the 15 of their papers we have counts for
17 papers
LLM-jp-4 Technical Report
LLM-jp, :, Chaoran Liu +16
This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-…
Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
Kotaro Yoshida, So Kuroki, Yuki Imajuku +4
Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organiz…
On the Optimal Reasoning Length for RL-Trained Language Models
Daisuke Nohara, Taishi Nakamura, Rio Yokota
Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although l…
MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
Huu Nguyen, Victor May, Harsh Raj +14
We present MixtureVitae, an open-access pretraining corpus built to minimize legal risk while providing strong downstream performance. MixtureVitae follows a permissive-first, risk…
Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Marianna Nezhurina, Harsh Raj, Jörg Franke +7
Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproduc…
Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura +4
Empirical scaling laws have driven the evolution of large language models (LLMs), yet their coefficients shift whenever the model architecture or data pipeline changes. Mixture-of-…
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Kazuki Fujii, Yukito Tajima, Sakae Mizuki +14
The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce…
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
Taishi Nakamura, Takuya Akiba, Kazuki Fujii +3
The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that init…
Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
Youmi Ma, Sakae Mizuki, Kazuki Fujii +12
Instruction tuning is crucial for enabling Large Language Models (LLMs) to solve real-world tasks. Prior work has shown the effectiveness of instruction-tuning data synthesized sol…
Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search
Yuichi Inoue, Kou Misaki, Yuki Imajuku +3
Recent advances demonstrate that increasing inference-time computation can significantly boost the reasoning capabilities of large language models (LLMs). Although repeated samplin…
Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs
Koshiro Saito, Sakae Mizuki, Masanari Ohi +11
Why do we build local large language models (LLMs)? What should a local LLM learn from the target language? Which abilities can be transferred from other languages? Do language-spe…
Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
Kazuki Fujii, Taishi Nakamura, Rio Yokota
Large Language Models (LLMs) have attracted significant attention due to their human-like language understanding and generation capabilities, as well as their applicability across…
Agent Skill Acquisition for Large Language Models via CycleQD
So Kuroki, Taishi Nakamura, Takuya Akiba +1
Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inad…
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
LLM-jp, :, Akiko Aizawa +80
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and stron…
Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
Kazuki Fujii, Taishi Nakamura, Mengsay Loem +7
Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and red…
Building a Large Japanese Web Corpus for Large Language Models
Naoaki Okazaki, Kakeru Hattori, Hirai Shota +7
Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the qua…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…