2 citations · 6 across the 19 of their papers we have counts for
22 papers
BCT: Bayes-Consistent Tempering for Flow Language Models
Daisuke Oba, Hiroki Furuta, Naoaki Okazaki
Temperature scaling is typically applied as a post-hoc transformation of model probabilities. In posterior-induced flow language models, this creates two mismatches. First, posteri…
Beyond Model Weights: Sampling-Dependent Fairness in Diffusion Language Models
Koshiro Saito, Daisuke Oba, Diasuke Oba +2
Diffusion language models (DLMs) are emerging as a practical alternative to autoregressive language models (ARLMs), yet their fairness remains largely uncharacterized. DLMs introdu…
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Sangwhan Moon, Daisuke Oba, Youmi Ma +2
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investi…
Neuron Level Analysis of Large Language Model in Legal Domain Reasoning
Eri Onami, Youmi Ma, Shuhei Kurita +1
We presented a neuron-level analysis of legal-domain reasoning in LLMs, comparing it with other applied domain tasks across seven open-weight models. Using neuron attribution score…
Drifting Objectives for Refining Discrete Diffusion Language Models
Daisuke Oba, Hiroki Furuta, Naoaki Okazaki
Discrete diffusion language models (DDLMs) generate text by iteratively denoising categorical token sequences, while recent drifting methods for continuous generators suggest that…
LLM Output Detectability and Task Performance Can be Jointly Optimized
Koshiro Saito, Ryuto Koike, Masahiro Kaneko +1
Detecting machine-generated text is essential for transparency and accountability when deploying LLMs. Watermarking enables statistically reliable detection by biasing token distri…
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Tatsuya Ichinose, Youmi Ma, Masanari Oi +2
Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a…
Aligning Tree-Search Policies with Fixed Token Budgets in Test-Time Scaling of LLMs
Sora Miyamoto, Daisuke Oba, Naoaki Okazaki
Tree-search decoding is an effective form of test-time scaling for large language models (LLMs), but real-world deployment often imposes a fixed per-query token budget that varies…
Autoregressive Direct Preference Optimization
Masanari Oi, Mahiro Ukai, Masahiro Kaneko +2
Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences. However, the widespread reliance on the r…
Diffusion-State Policy Optimization for Masked Diffusion Language Models
Daisuke Oba, Hiroki Furuta, Naoaki Okazaki
Masked diffusion language models generate text through iterative masked-token filling, but terminal-only rewards on final completions provide coarse credit assignment for the inter…
Stopping Computation for Converged Tokens in Masked Diffusion-LM Decoding
Daisuke Oba, Danushka Bollegala, Masahiro Kaneko +1
Masked Diffusion Language Models generate sequences via iterative sampling that progressively unmasks tokens. However, they still recompute the attention and feed-forward blocks fo…
From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
Youmi Ma, Naoaki Okazaki
Advances in mechanistic interpretability have identified special attention heads, known as retrieval heads, that are responsible for retrieving information from the context. Howeve…
QuantumBench: A Benchmark for Quantum Problem Solving
Shunya Minami, Tatsuya Ishigaki, Ikko Hamamura +6
Large language models are now integrated into many scientific workflows, accelerating data analysis, hypothesis generation, and design space exploration. In parallel with this grow…
Machine Text Detectors are Membership Inference Attacks
Ryuto Koike, Liam Dugan, Masahiro Kaneko +2
Although membership inference attacks (MIAs) and machine-generated text detection target different goals, their methods often exploit similar signals based on a language model's pr…
Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Models
Panatchakorn Anantaprayoon, Masahiro Kaneko, Naoaki Okazaki
Self-Correction based on feedback improves the output quality of Large Language Models (LLMs). Moreover, as Self-Correction functions like the slow and conscious System-2 thinking…
ExaGPT: Example-Based Machine-Generated Text Detection for Human Interpretability
Ryuto Koike, Masahiro Kaneko, Ayana Niwa +2
Detecting texts generated by Large Language Models (LLMs) could cause grave mistakes due to incorrect decisions, such as undermining students' academic dignity. LLM text detection…
HMoE: Heterogeneous Mixture of Experts for Language Modeling
An Wang, Xingwu Sun, Ruobing Xie +9
Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous…
Social Bias Evaluation for Large Language Models Requires Prompt Variations
Rem Hida, Masahiro Kaneko, Naoaki Okazaki
Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mit…
Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer
Youmi Ma, An Wang, Naoaki Okazaki
Document-level Relation Extraction (DocRE) is the task of extracting all semantic relationships from a document. While studies have been conducted on English DocRE, limited attenti…
Sampling-based Pseudo-Likelihood for Membership Inference Attacks
Masahiro Kaneko, Youmi Ma, Yuki Wata +1
Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data…
An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki +2
We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords. The technique…
Knowledge of Pretrained Language Models on Surface Information of Tokens
Tatsuya Hiraoka, Naoaki Okazaki
Do pretrained language models have knowledge regarding the surface information of tokens? We examined the surface information stored in word or subword embeddings acquired by pretr…