24 citations · 41 across the 5 of their papers we have counts for
7 papers · 1 filter
Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes
Kosuke Nishida, Kyosuke Nishida, Kuniko Saito
Loss spikes, a phenomenon in which the loss value diverges suddenly, is a fundamental issue in the pre-training of large language models. This paper supposes that the non-uniformit…
Task-adaptive Pre-training of Language Models with Word Embedding Regularization
Kosuke Nishida, Kyosuke Nishida, Sen Yoshida
Pre-trained language models (PTLMs) acquire domain-independent linguistic knowledge through pre-training with massive textual resources. Additional pre-training is effective in ada…
Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida +1
Pre-trained sequence-to-sequence (seq-to-seq) models have significantly improved the accuracy of several language generation tasks, including abstractive summarization. Although th…
Length-controllable Abstractive Summarization by Guiding with Summary Prototype
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida +5
We propose a new length-controllable abstractive summarization model. Recent state-of-the-art abstractive summarization models based on encoder-decoder models generate only one sum…
Unsupervised Domain Adaptation of Language Models for Reading Comprehension
Kosuke Nishida, Kyosuke Nishida, Itsumi Saito +2
This study tackles unsupervised domain adaptation of reading comprehension (UDARC). Reading comprehension (RC) is a task to learn the capability for question answering with textual…
Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata +4
Question answering (QA) using textual sources for purposes such as reading comprehension (RC) has attracted much attention. This study focuses on the task of explainable multi-hop…