114 citations · 355 across the 20 of their papers we have counts for
5 papers · 1 filter
Head Dimension Matters in Language Models for Long-Context Retrieval
Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi
Multi-head attention retrieves and combines context representations through query--key interactions and value aggregation, with head dimension constraining the per-head representat…
B2T Connection: Serving Stability and Performance in Deep Transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi +1
From the perspective of the layer normalization (LN) positions, the architectures of Transformers can be categorized into two types: Post-LN and Pre-LN. Recent Transformers tend to…
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki +1
Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effectiv…
Efficient Estimation of Influence of a Training Instance
Sosuke Kobayashi, Sho Yokoi, Jun Suzuki +1
Understanding the influence of a training instance on a neural network model leads to improving interpretability. However, it is difficult and inefficient to evaluate the influence…
Data Interpolating Prediction: Alternative Interpretation of Mixup
Takuya Shimada, Shoichiro Yamaguchi, Kohei Hayashi +1
Data augmentation by mixing samples, such as Mixup, has widely been used typically for classification tasks. However, this strategy is not always effective due to the gap between a…