Diminishing Returns in Self-Supervised Learning

Oli Bridge, Huey Sun, Botond Branyicskai-Nagy +2

Transformer-based architectures have become a dominant paradigm in vision and language, but their success is often attributed to large model capacity and massive training data. In…