SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation

Abstract

This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantically meaningful text representation as an initial embedding in the autoregressive training and generation processes. The extensive empirical study demonstrates that the SMCLM approach makes autoregressive models capable of learning robust and high-quality paraphrase generation. The proposed method is competitive with the supervised method and achieves state-of-the-art results in unsupervised approaches. This article also presents a comprehensive set of automatic metrics that cover a wide range of autogenerated paraphrase evaluation aspects. Simultaneously, this article highlights the low reliability of the metrics that are widely used in paraphrase generation evaluation, including BLEU, ROUGE, and BERTScore.

References in corpus (11)

• Universal Sentence Encoder
• Self-supervised Learning: Generative or Contrastive
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• Language-agnostic BERT Sentence Embedding
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• Efficient generative modeling of protein sequences using simple autoregressive models
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• Unsupervised Paraphrase Generation using Pre-trained Language Models
• Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents
• Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning