How Redundant Is the Transformer Stack in Speech Representation Models?

Teresa Dorszewski, Albert Kjøller Jacobsen, Lenka Tětková +1

Self-supervised speech representation models, particularly those leveraging transformer architectures, have demonstrated remarkable performance across various tasks such as speech…