Direct speech-to-speech translation with a sequence-to-sequence model
arXiv:1904.06037
Abstract
We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.
Accepted to Interspeech 2019
References in corpus (7)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
- Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
- Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation
- On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
- Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
- End-to-End Spoken Language Translation