24 citations · 73 across the 19 of their papers we have counts for
20 papers
North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)
Tom Kocmi, Alexandre Bérard, Phil Blunsom +20
We present North Small Translate, an open-weight, LLM-based machine translation (MT) model with instruction-following capabilities built on the same foundation as Cohere's Command…
Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Kazuki Yano, Shun Kiyono, Sosuke Kobayashi +2
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised f…
Efficient Construction of Model Family through Progressive Training Using Model Expansion
Kazuki Yano, Sho Takase, Sosuke Kobayashi +2
As Large Language Models (LLMs) gain widespread practical application, offering model families with varying parameter sizes has become standard practice to accommodate diverse comp…
Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability
Ryokan Ri, Shun Kiyono, Sho Takase
Zero-shot cross-lingual transfer by fine-tuning multilingual pretrained models shows promise for low-resource languages, but often suffers from misalignment of internal representat…
Large Vocabulary Size Improves Large Language Models
Sho Takase, Ryokan Ri, Shun Kiyono +1
This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vo…
Spike No More: Stabilizing the Pre-training of Large Language Models
Sho Takase, Shun Kiyono, Sosuke Kobayashi +1
Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre…
B2T Connection: Serving Stability and Performance in Deep Transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi +1
From the perspective of the layer normalization (LN) positions, the architectures of Transformers can be categorized into two types: Post-LN and Pre-LN. Recent Transformers tend to…
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki +1
Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effectiv…
SHAPE: Shifted Absolute Position Embedding for Transformers
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki +1
Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data…
Rethinking Perturbations in Encoder-Decoders for Fast Training
Sho Takase, Shun Kiyono
We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbat…
Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution
Ryuto Konno, Shun Kiyono, Yuichiroh Matsubayashi +2
Masked language models (MLMs) have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). To further improve this approach, in this study, w…
Lessons on Parameter Sharing across Layers in Transformers
Sho Takase, Shun Kiyono
We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all…
An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution
Ryuto Konno, Yuichiroh Matsubayashi, Shun Kiyono +3
One critical issue of zero anaphora resolution (ZAR) is the scarcity of labeled data. This study explores how effectively this problem can be alleviated by data augmentation. We ad…
A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction
Masato Mita, Shun Kiyono, Masahiro Kaneko +2
Existing approaches for grammatical error correction (GEC) largely rely on supervised learning with manually created GEC datasets. However, there has been little focus on verifying…
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
Masahiro Kaneko, Masato Mita, Shun Kiyono +2
This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correct…
ESPnet-ST: All-in-One Speech Translation Toolkit
Hirofumi Inaguma, Shun Kiyono, Kevin Duh +4
We present ESPnet-ST, which is designed for the quick development of speech-to-speech translation systems in a single framework. ESPnet-ST is a new project inside end-to-end speech…
Riposte! A Large Corpus of Counter-Arguments
Paul Reisert, Benjamin Heinzerling, Naoya Inoue +2
Constructive feedback is an effective method for improving critical thinking skills. Counter-arguments (CAs), one form of constructive feedback, have been proven to be useful for c…
An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction
Shun Kiyono, Jun Suzuki, Masato Mita +2
The incorporation of pseudo data in the training of grammatical error correction models has been one of the main factors in improving the performance of such models. However, conse…
Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
Shun Kiyono, Jun Suzuki, Kentaro Inui
The current success of deep neural networks (DNNs) in an increasingly broad range of tasks involving artificial intelligence strongly depends on the quality and quantity of labeled…
Source-side Prediction for Neural Headline Generation
Shun Kiyono, Sho Takase, Jun Suzuki +3
The encoder-decoder model is widely used in natural language generation tasks. However, the model sometimes suffers from repeated redundant generation, misses important phrases, an…