24 citations · 73 across the 19 of their papers we have counts for
4 papers · 1 filter
SHAPE: Shifted Absolute Position Embedding for Transformers
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki +1
Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data…
Rethinking Perturbations in Encoder-Decoders for Fast Training
Sho Takase, Shun Kiyono
We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbat…
Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution
Ryuto Konno, Shun Kiyono, Yuichiroh Matsubayashi +2
Masked language models (MLMs) have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). To further improve this approach, in this study, w…
Lessons on Parameter Sharing across Layers in Transformers
Sho Takase, Shun Kiyono
We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The proposed approach relaxes a widely used technique, which shares parameters for one layer with all…