20 citations · 20 across the 2 of their papers we have counts for
5 papers
On Optimal Transformer Depth for Low-Resource Language Translation
Elan van Biljon, Arnu Pretorius, Julia Kreutzer
Transformers have shown great promise as an approach to Neural Machine Translation (NMT) for low-resource languages. However, at the same time, transformer models remain difficult…
Masakhane -- Machine Translation For Africa
Iroro Orife, Julia Kreutzer, Blessing Sibanda +22
Africa has over 2000 languages. Despite this, African languages account for a small portion of available resources and publications in Natural Language Processing (NLP). This is du…
If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
Arnu Pretorius, Elan van Biljon, Benjamin van Niekerk +6
Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effe…
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks
Ryan Eloff, André Nortje, Benjamin van Niekerk +7
For our submission to the ZeroSpeech 2019 challenge, we apply discrete latent-variable neural networks to unlabelled speech and use the discovered units for speech synthesis. Unsup…
Critical initialisation for deep signal propagation in noisy rectifier neural networks
Arnu Pretorius, Elan Van Biljon, Steve Kroon +1
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influe…