8 papers · 1 filter
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Sangwhan Moon, Daisuke Oba, Youmi Ma +2
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investi…
Neuron Level Analysis of Large Language Model in Legal Domain Reasoning
Eri Onami, Youmi Ma, Shuhei Kurita +1
We presented a neuron-level analysis of legal-domain reasoning in LLMs, comparing it with other applied domain tasks across seven open-weight models. Using neuron attribution score…
From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
Youmi Ma, Naoaki Okazaki
Advances in mechanistic interpretability have identified special attention heads, known as retrieval heads, that are responsible for retrieving information from the context. Howeve…
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Tatsuya Ichinose, Youmi Ma, Masanari Oi +2
Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a…
Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs
Koshiro Saito, Sakae Mizuki, Masanari Ohi +11
Why do we build local large language models (LLMs)? What should a local LLM learn from the target language? Which abilities can be transferred from other languages? Do language-spe…
Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
Youmi Ma, Sakae Mizuki, Kazuki Fujii +12
Instruction tuning is crucial for enabling Large Language Models (LLMs) to solve real-world tasks. Prior work has shown the effectiveness of instruction-tuning data synthesized sol…