2 citations · 10 across the 18 of their papers we have counts for
19 papers
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3
Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical pri…
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos Aletras
Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this iss…
On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
Atsuki Yamaguchi, Szymon Palucha, Léo Bijar +2
Mixture-of-Experts (MoE) models offer inference speedups via selective activation but impose substantial memory requirements because the whole network must remain loaded. Structure…
Rethinking the Idiomaticity Decomposability Hypothesis: Evidence from Distributional Learning
Maggie Mi, Golzar Atefi, Atsuki Yamaguchi +3
Idioms can be analysed in terms of their decomposability, the extent to which constituent meanings contribute to the figurative whole. Decomposability is thought to predict syntact…
MultiHashFormer: Hash-based Generative Language Models
Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras
Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many…
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9
Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and s…
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
Atsuki Yamaguchi, Maggie Mi, Nikolaos Aletras
Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning,…
Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates
Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio +1
Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target l…
Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi +1
Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning. To address this, we propose $\textbf{Additional Logic Training (A…
Adapting Chat Language Models Using Only Target Unlabeled Language Data
Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio +1
Vocabulary expansion (VE) is the de-facto approach to language adaptation of large language models (LLMs) by adding new tokens and continuing pre-training on target data. While thi…
How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?
Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras
Large language models (LLMs) have shown remarkable capabilities in many languages beyond English. Yet, LLMs require more inference steps when generating non-English text due to the…
An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference
Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras
The development of state-of-the-art generative large language models (LLMs) disproportionately relies on English-centric tokenizers, vocabulary and pre-training data. Despite the f…
appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkit
Atsuki Yamaguchi, Terufumi Morishita
We present appjsonify, a Python-based PDF-to-JSON conversion toolkit for academic papers. It parses a PDF file using several visual-based document layout analysis models and rule-b…
Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi +1
We study a synthetic corpus based approach for language models (LMs) to acquire logical deductive reasoning ability. The previous studies generated deduction examples using specifi…
How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese
Takuro Fujii, Koki Shibata, Atsuki Yamaguchi +2
This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist be…
How does the task complexity of masked pretraining objectives affect downstream performance?
Atsuki Yamaguchi, Hiroaki Ozaki, Terufumi Morishita +2
Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts.…
Hitachi at SemEval-2023 Task 3: Exploring Cross-lingual Multi-task Strategies for Genre and Framing Detection in Online News
Yuta Koreeda, Ken-ichi Yokote, Hiroaki Ozaki +3
This paper explains the participation of team Hitachi to SemEval-2023 Task 3 "Detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setu…
Team Hitachi @ AutoMin 2021: Reference-free Automatic Minuting Pipeline with Argument Structure Construction over Topic-based Summarization
Atsuki Yamaguchi, Gaku Morio, Hiroaki Ozaki +2
This paper introduces the proposed automatic minuting system of the Hitachi team for the First Shared Task on Automatic Minuting (AutoMin-2021). We utilize a reference-free approac…
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling
Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina +1
Masked language modeling (MLM), a self-supervised pretraining objective, is widely used in natural language processing for learning text representations. MLM trains a model to pred…