Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

Rasmus Aagaard, Nicki Skafte Detlefsen

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups…