5 papers · 1 filter
Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
Tzu-Quan Lin, Tsung-Huan Yang, Chun-Yao Chang +4
Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real…
Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC
Shen-sian Syu, Juncheng Xie, Hung-yi Lee
Non-autoregressive approaches aim to improve the inference speed of translation models, particularly those that generate output in a one-pass forward manner. However, these approac…
MelHuBERT: A simplified HuBERT on Mel spectrograms
Tzu-Quan Lin, Hung-yi Lee, Hao Tang
Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks. However, most self-supervised models require a lar…
Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning
Shih-Cheng Huang, Shih-Heng Wang, Min-Han Shih +2
Parameter-efficient (PE) methods (like Prompts or Adapters) for adapting pre-trained language models (PLM) to downstream tasks have been popular recently. However, hindrances still…
How to Estimate Model Transferability of Pre-Trained Speech Models?
Zih-Ching Chen, Chao-Han Huck Yang, Bo Li +6
In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upo…