Empirical Capacity Model for Self-Attention Neural Networks

Aki Härmä, Marcin Pietrasik, Anna Wilbik

Large pretrained self-attention neural networks, or transformers, have been very successful in various tasks recently. The performance of a model on a given task depends on its abi…