The Daunting Task of Real-World Textual Style Transfer Auto-Evaluation

Abstract

The difficulty of textual style transfer lies in the lack of parallel corpora. Numerous advances have been proposed for the unsupervised generation. However, significant problems remain with the auto-evaluation of style transfer tasks. Based on the summary of Pang and Gimpel (2018) and Mir et al. (2019), style transfer evaluations rely on three criteria: style accuracy of transferred sentences, content similarity between original and transferred sentences, and fluency of transferred sentences. We elucidate the problematic current state of style transfer research. Given that current tasks do not represent real use cases of style transfer, current auto-evaluation approach is flawed. This discussion aims to bring researchers to think about the future of style transfer and style transfer evaluation research.

arXiv commentExtended abstract in EMNLP Workshop on Neural Generation and Translation (WNGT 2019)

References in corpus (5)

• Content preserving text generation with attribute controls
• Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
• Unsupervised Text Style Transfer using Language Models as Discriminators
• Evaluating Style Transfer for Text
• $A^{4}NT$: Author Attribute Anonymity by Adversarial Training of Neural Machine Translation

Cited by in corpus (3)

• From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey
• Review of Text Style Transfer Based on Deep Learning
• Investigating Text Simplification Evaluation