Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric

Abstract

The rapid development of such natural language processing tasks as style transfer, paraphrase, and machine translation often calls for the use of semantic similarity metrics. In recent years a lot of methods to measure the semantic similarity of two short texts were developed. This paper provides a comprehensive analysis for more than a dozen of such methods. Using a new dataset of fourteen thousand sentence pairs human-labeled according to their semantic similarity, we demonstrate that none of the metrics widely used in the literature is close enough to human judgment in these tasks. A number of recently proposed metrics provide comparable results, yet Word Mover Distance is shown to be the most reasonable solution to measure semantic similarity in reformulated texts at the moment.

References in corpus (5)

• BERTScore: Evaluating Text Generation with BERT
• Neural Paraphrase Generation with Stacked Residual LSTM Networks
• A Deep Generative Framework for Paraphrase Generation
• Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites
• Decomposing Textual Information For Style Transfer

Cited by in corpus (5)

• Review of Text Style Transfer Based on Deep Learning
• What is Wrong with Language Models that Can Not Tell a Story?
• Studying the role of named entities for content preservation in text style transfer
• Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation