Direct Speech-to-image Translation
arXiv:2004.03413 · doi:10.1109/JSTSP.2020.2987417
Abstract
Direct speech-to-image translation without text is an interesting and useful topic due to the potential applications in human-computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In this paper, we attempt to translate the speech signals into the image signals without the transcription stage. Specifically, a speech encoder is designed to represent the input speech signals as an embedding feature, and it is trained with a pretrained image encoder using teacher-student learning to obtain better generalization ability on new classes. Subsequently, a stacked generative adversarial network is used to synthesize high-quality images conditioned on the embedding feature. Experimental results on both synthesized and real data show that our proposed method is effective to translate the raw speech signals into images without the middle text representation. Ablation study gives more insights about our method.
Accepted by JSTSP
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Conditional Generative Adversarial Nets
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Generative Adversarial Text to Image Synthesis
- Do Deep Nets Really Need to be Deep?
- The Microsoft 2017 Conversational Speech Recognition System
- Deep Voice: Real-time Neural Text-to-Speech
- TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
- Deep Voice 2: Multi-Speaker Neural Text-to-Speech
- Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
- End-to-End ASR-free Keyword Search from Speech