Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translation

Abstract

Multimodal large language models (MLLMs) have demonstrated significant potential for speech-to-text translation (S2TT). However, existing deployment paradigms face critical challenges: pure on-device models suffer from resource constraints, while centralized cloud systems incur bandwidth bottlenecks and privacy risks by transmitting raw voice data. In this paper, we propose Edge--cloud Speech Recognition and Translation (ESRT), a parameter-efficient, bandwidth-efficient, and privacy-aware collaborative Edge--cloud MLLM framework. First, we introduce a multi-task weighted curriculum learning strategy to mitigate catastrophic forgetting, improve multilingual balance, and train parameter-efficient ESRT-1B, ESRT-4B, and ESRT-12B models. Second, we enable bandwidth-efficient Edge--cloud inference by retaining a lightweight speech encoder and adapter on the device and transmitting only a compressed tensor to the cloud. Extensive experiments on FLEURS demonstrate that ESRT models achieve state-of-the-art S2TT performance across 45 languages ($45\times 44$ directions). Relative to raw audio, ESRT and ESRT-Lite reduce the transmitted tensor size by $5.1\times$ and $10.2\times$, respectively, while keeping raw speech on-device and avoiding its direct exposure to the cloud. The code and models are released to facilitate reproducible, privacy-aware S2TT research.