Human Language Modeling

Abstract

Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem whereby a human-level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for the HuLM task, pre-trained on approximately 100,000 social media users, and demonstrate its effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels: stance detection, sentiment classification, age estimation, and personality assessment. Results on all tasks meet or surpass the current state-of-the-art.

References in corpus (6)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Longformer: The Long-Document Transformer
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• REALM: Retrieval-Augmented Language Model Pre-Training
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size