Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark

Abstract

Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms including conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMs understand \textit{social} language. Here, we introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge which we group into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. In tests on the benchmark, we demonstrate that current models attain only moderate performance but reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero-shot evaluations, we show that pretrained models already possess some innate but limited capabilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides a systematic way to analyze model performance on an important dimension of language and points to clear room for improvement to build more socially-aware LLMs. The associated resources are released at https://github.com/minjechoi/SOCKET.

arXiv commentCamera-ready version for EMNLP'23. First two authors contributed equally

References in corpus (15)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Scaling Instruction-Finetuned Language Models
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• A Large Self-Annotated Corpus for Sarcasm
• Dissociating language and thought in large language models
• SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles
• Utterance-level Dialogue Understanding: An Empirical Study
• MOVER: Mask, Over-generate and Rank for Hyperbole Generation

Cited by in corpus (4)

• A Survey on Large Language Model based Autonomous Agents
• Exploring the applicability of Large Language Models to citation context analysis
• On Sarcasm Detection with OpenAI GPT-based Models
• Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengali