Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency

Sungho Jeon, Ching-Feng Yeh, Hakan Inan +4

In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transfor…