How to Estimate Model Transferability of Pre-Trained Speech Models?

Abstract

In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates using the extracted representations. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers by making a temporal independent hypothesis. We evaluate some popular supervised speech models (e.g., Conformer RNN-Transducer) and self-supervised speech models (e.g., HuBERT) in cross-layer and cross-model settings using public data. Experimental results show a high Spearman's rank correlation and low $p$-value between our estimation framework and fine-tuning ground truth. Our proposed transferability framework requires less computational time and resources, making it a resource-saving and time-efficient approach for tuning speech foundation models.

Accepted to Interspeech. Code is available at: https://github.com/virginiakm1988/LogME-CTC. Fixed a typo

References in corpus (12)

• On the Opportunities and Risks of Foundation Models
• Sequence Transduction with Recurrent Neural Networks
• Robust Speech Recognition via Large-Scale Weak Supervision
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• MLS: A Large-Scale Multilingual Dataset for Speech Research
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• LEEP: A New Measure to Evaluate Transferability of Learned Representations
• Geometric Dataset Distances via Optimal Transport
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Device-Robust Acoustic Scene Classification Based on Two-Stage Categorization and Data Augmentation
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition

Cited by in corpus (2)

• Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
• Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition