SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding

Abstract

We study the ability of transformer-based language models (LMs) to understand social media language. Social media (SM) language is distinct from standard written language, yet existing benchmarks fall short of capturing LM performance in this socially, economically, and politically important domain. We quantify the degree to which social media language differs from conventional language and conclude that the difference is significant both in terms of token distribution and rate of linguistic shift. Next, we introduce a new benchmark for Social MedIa Language Evaluation (SMILE) that covers four SM platforms and eleven tasks. Finally, we show that learning a tokenizer and pretraining on a mix of social media and conventional language yields an LM that outperforms the best similar-sized alternative by 4.2 points on the overall SMILE score.

References in corpus (22)

• Character-level Convolutional Networks for Text Classification
• On the Opportunities and Risks of Foundation Models
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• PaLM: Scaling Language Modeling with Pathways
• Scaling Laws for Neural Language Models
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• LaMDA: Language Models for Dialog Applications
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Hyena Hierarchy: Towards Larger Convolutional Language Models
• ByT5: Towards a token-free future with pre-trained byte-to-byte models
• Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
• Scaling Up Models and Data with $\text{t5x}$ and $\text{seqio}$
• Dynamic Language Models for Continuously Evolving Content
• TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter
• Stochastic Learning for Sparse Discrete Markov Random Fields with Controlled Gradient Approximation Error
• An Efficient Pseudo-likelihood Method for Sparse Binary Pairwise Markov Network Estimation
• Partially Linear Additive Gaussian Graphical Models
• Joint Nonparametric Precision Matrix Estimation with Confounding