24 citations · 76 across the 25 of their papers we have counts for
25 papers
Toward LLMs Beyond English-Centric Development
Sho Takase, Ukyo Honda
Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is…
Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Kazuki Yano, Shun Kiyono, Sosuke Kobayashi +2
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised f…
Efficient Construction of Model Family through Progressive Training Using Model Expansion
Kazuki Yano, Sho Takase, Sosuke Kobayashi +2
As Large Language Models (LLMs) gain widespread practical application, offering model families with varying parameter sizes has become standard practice to accommodate diverse comp…
Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability
Ryokan Ri, Shun Kiyono, Sho Takase
Zero-shot cross-lingual transfer by fine-tuning multilingual pretrained models shows promise for low-resource languages, but often suffers from misalignment of internal representat…
Large Vocabulary Size Improves Large Language Models
Sho Takase, Ryokan Ri, Shun Kiyono +1
This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vo…
Spike No More: Stabilizing the Pre-training of Large Language Models
Sho Takase, Shun Kiyono, Sosuke Kobayashi +1
Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre…
Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods
Mengsay Loem, Masahiro Kaneko, Sho Takase +1
Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks. However, applying prompt-based methods wit…
Nearest Neighbor Non-autoregressive Text Generation
Ayana Niwa, Sho Takase, Naoaki Okazaki
Non-autoregressive (NAR) models can generate sentences with less computation than autoregressive models but sacrifice generation quality. Previous studies addressed this issue thro…
Are Neighbors Enough? Multi-Head Neural n-gram can be Alternative to Self-attention
Mengsay Loem, Sho Takase, Masahiro Kaneko +1
Impressive performance of Transformer has been attributed to self-attention, where dependencies between entire input in a sequence are considered at every position. In this work, w…
B2T Connection: Serving Stability and Performance in Deep Transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi +1
From the perspective of the layer normalization (LN) positions, the architectures of Transformers can be categorized into two types: Post-LN and Pre-LN. Recent Transformers tend to…
Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
Sho Takase, Tatsuya Hiraoka, Naoaki Okazaki
Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we u…
Interpretability for Language Learners Using Example-Based Grammatical Error Correction
Masahiro Kaneko, Sho Takase, Ayana Niwa +1
Grammatical Error Correction (GEC) should not focus only on high accuracy of corrections but also on interpretability for language learning. However, existing neural-based GEC mode…
ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization
Mengsay Loem, Sho Takase, Masahiro Kaneko +1
Neural models trained with large amount of parallel data have achieved impressive performance in abstractive summarization tasks. However, large-scale parallel corpora are expensiv…
Joint Optimization of Tokenization and Downstream Model
Tatsuya Hiraoka, Sho Takase, Kei Uchiumi +2
Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies…
Rethinking Perturbations in Encoder-Decoders for Fast Training
Sho Takase, Shun Kiyono
We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbat…
Lessons on Parameter Sharing across Layers in Transformers
Sho Takase, Shun Kiyono
We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all…
Multi-Task Learning for Cross-Lingual Abstractive Summarization
Sho Takase, Naoaki Okazaki
We present a multi-task learning framework for cross-lingual abstractive summarization to augment training data. Recent studies constructed pseudo cross-lingual abstractive summari…
Improving Truthfulness of Headline Generation
Kazuki Matsumaru, Sho Takase, Naoaki Okazaki
Most studies on abstractive summarization report ROUGE scores between system and reference summaries. However, we have a concern about the truthfulness of generated summaries: whet…
All Word Embeddings from One Embedding
Sho Takase, Sosuke Kobayashi
In neural network-based models for natural language processing (NLP), the largest part of the parameters often consists of word embeddings. Conventional models prepare a large embe…
Character n-gram Embeddings to Improve RNN Language Models
Sho Takase, Jun Suzuki, Masaaki Nagata
This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the fiel…
Positional Encoding to Control Output Sequence Length
Sho Takase, Naoaki Okazaki
Neural encoder-decoder models have been successful in natural language generation tasks. However, real applications of abstractive summarization must consider additional constraint…
Direct Output Connection for a High-Rank Language Model
Sho Takase, Jun Suzuki, Masaaki Nagata
This paper proposes a state-of-the-art recurrent neural network (RNN) language model that combines probability distributions computed not only from a final RNN layer but also from…
Source-side Prediction for Neural Headline Generation
Shun Kiyono, Sho Takase, Jun Suzuki +3
The encoder-decoder model is widely used in natural language generation tasks. However, the model sometimes suffers from repeated redundant generation, misses important phrases, an…
Input-to-Output Gate to Improve RNN Language Models
Sho Takase, Jun Suzuki, Masaaki Nagata
This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Outp…