STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models

Kangwook Jang, Sungnyun Kim, Hoirin Kim

Albeit great performance of Transformer-based speech selfsupervised learning (SSL) models, their large parameter size and computational cost make them unfavorable to utilize. In th…