114 citations · 355 across the 20 of their papers we have counts for
12 papers · 1 filter
Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Kazuki Yano, Shun Kiyono, Sosuke Kobayashi +2
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised f…
When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa +7
The ability to acquire latent semantics is one of the key properties that determines the performance of language models. One convenient approach to invoke this ability is to prepen…
Efficient Construction of Model Family through Progressive Training Using Model Expansion
Kazuki Yano, Sho Takase, Sosuke Kobayashi +2
As Large Language Models (LLMs) gain widespread practical application, offering model families with varying parameter sizes has become standard practice to accommodate diverse comp…
Spike No More: Stabilizing the Pre-training of Large Language Models
Sho Takase, Shun Kiyono, Sosuke Kobayashi +1
Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre…
Instance-Based Neural Dependency Parsing
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi +4
Interpretable rationales for model predictions are crucial in practical applications. We develop neural models that possess an interpretable inference process for dependency parsin…
SHAPE: Shifted Absolute Position Embedding for Transformers
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki +1
Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data…
Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi +4
Interpretable rationales for model predictions play a critical role in practical applications. In this study, we develop models possessing interpretable inference process for struc…
All Word Embeddings from One Embedding
Sho Takase, Sosuke Kobayashi
In neural network-based models for natural language processing (NLP), the largest part of the parameters often consists of word embeddings. Conventional models prepare a large embe…
Pointwise HSIC: A Linear-Time Kernelized Co-occurrence Norm for Sparse Linguistic Expressions
Sho Yokoi, Sosuke Kobayashi, Kenji Fukumizu +2
In this paper, we propose a new kernel-based co-occurrence measure that can be applied to sparse linguistic expressions (e.g., sentences) with a very short learning time, as an alt…
Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations
Sosuke Kobayashi
We propose a novel data augmentation for labeled sentences called contextual augmentation. We assume an invariance that sentences are natural even if the words in the sentences are…
Unsupervised Learning of Style-sensitive Word Vectors
Reina Akama, Kento Watanabe, Sho Yokoi +2
This paper presents the first study aimed at capturing stylistic similarity between words in an unsupervised manner. We propose extending the continuous bag of words (CBOW) model (…
A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
Sosuke Kobayashi, Naoaki Okazaki, Kentaro Inui
This study addresses the problem of identifying the meaning of unknown words or entities in a discourse with respect to the word embedding approaches used in neural language models…