Re-Evaluating GermEval17 Using German Pre-Trained Language Models

Abstract

The lack of a commonly used benchmark data set (collection) such as (Super-)GLUE (Wang et al., 2018, 2019) for the evaluation of non-English pre-trained language models is a severe shortcoming of current English-centric NLP-research. It concentrates a large part of the research on English, neglecting the uncertainty when transferring conclusions found for the English language to other languages. We evaluate the performance of the German and multilingual BERT-based models currently available via the huggingface transformers library on the four tasks of the GermEval17 workshop. We compare them to pre-BERT architectures (Wojatzki et al., 2017; Schmitt et al., 2018; Attia et al., 2018) as well as to an ELMo-based architecture (Biesialska et al., 2020) and a BERT-based approach (Guhr et al., 2020). The observed improvements are put in relation to those for similar tasks and similar models (pre-BERT vs. BERT-based) for the English language in order to draw tentative conclusions about whether the observed improvements are transferable to German or potentially other related languages.

arXiv commentAccepted as a conference paper at the 6th Swiss Text Analytics Conference (SwissText), Brugg, Switzerland (Online), June 14-16, 2021

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
• Learned in Translation: Contextualized Word Vectors
• Aspect Level Sentiment Classification with Deep Memory Network
• ner and pos when nothing is capitalized
• A Multi-task Learning Model for Chinese-oriented Aspect Polarity Classification and Aspect Term Extraction
• Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis
• Sentiment Analysis with Contextual Embeddings and Self-Attention