Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training

Abstract

Automatic text simplification systems help to reduce textual information barriers on the internet. However, for languages other than English, only few parallel data to train these systems exists. We propose a two-step approach to overcome this data scarcity issue. First, we fine-tuned language models on a corpus of German Easy Language, a specific style of German. Then, we used these models as decoders in a sequence-to-sequence simplification task. We show that the language models adapt to the style characteristics of Easy Language and output more accessible texts. Moreover, with the style-specific pre-training, we reduced the number of trainable parameters in text simplification models. Hence, less parallel data is sufficient for training. Our results indicate that pre-training on unaligned data can reduce the required parallel data while improving the performance on downstream tasks.

Accepted to ACL Findings 2023

References in corpus (10)

• PaLM: Scaling Language Modeling with Pathways
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Multilingual Denoising Pre-training for Neural Machine Translation
• A Contrastive Framework for Neural Text Generation
• mGPT: Few-Shot Learners Go Multilingual
• GPT-based Generation for Classical Chinese Poetry
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• Perplexity from PLM Is Unreliable for Evaluating Text Quality
• A New Aligned Simple German Corpus
• SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words