Automated essay scoring using efficient transformer-based language models

Abstract

Automated Essay Scoring (AES) is a cross-disciplinary effort involving Education, Linguistics, and Natural Language Processing (NLP). The efficacy of an NLP model in AES tests it ability to evaluate long-term dependencies and extrapolate meaning even when text is poorly written. Large pretrained transformer-based language models have dominated the current state-of-the-art in many NLP tasks, however, the computational requirements of these models make them expensive to deploy in practice. The goal of this paper is to challenge the paradigm in NLP that bigger is better when it comes to AES. To do this, we evaluate the performance of several fine-tuned pretrained NLP models with a modest number of parameters on an AES dataset. By ensembling our models, we achieve excellent results with fewer parameters than most pretrained transformer-based models.

11 pages, 1 figure, 3 tables

References in corpus (11)

• Distributed Representations of Words and Phrases and their Compositionality
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• Rethinking Attention with Performers
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Single Headed Attention RNN: Stop Thinking With Your Head
• Language models and Automated Essay Scoring