Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?
arXiv:2108.12275
Abstract
In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.
arXiv commentaccepted to RANLP 2021
References in corpus (9)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Self-Attention Generative Adversarial Networks
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Training Tips for the Transformer Model
- A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning
- Generative Adversarial Transformers
- Pay Attention when Required