Dialectometric analysis of language variation in Twitter

Abstract

In the last few years, microblogging platforms such as Twitter have given rise to a deluge of textual data that can be used for the analysis of informal communication between millions of individuals. In this work, we propose an information-theoretic approach to geographic language variation using a corpus based on Twitter. We test our models with tens of concepts and their associated keywords detected in Spanish tweets geolocated in Spain. We employ dialectometric measures (cosine similarity and Jensen-Shannon divergence) to quantify the linguistic distance on the lexical level between cells created in a uniform grid over the map. This can be done for a single concept or in the general case taking into account an average of the considered variants. The latter permits an analysis of the dialects that naturally emerge from the data. Interestingly, our results reveal the existence of two dialect macrovarieties. The first group includes a region-specific speech spoken in small towns and rural areas whereas the second cluster encompasses cities that tend to use a more uniform variety. Since the results obtained with the two different metrics qualitatively agree, our work suggests that social media corpora can be efficiently used for dialectometric analyses.

arXiv comment10 pages, 7 figures, 1 table. Accepted to VarDial 2017

References in corpus (4)

• Generative Adversarial Networks
• Diffusion of Lexical Change in Social Media
• Crowdsourcing Dialect Characterization through Twitter
• Bootstrap Methods for the Empirical Study of Decision-Making and Information Flows in Social Systems

Cited by in corpus (14)

• Mapping the Americanization of English in Space and Time
• American cultural regions mapped through the lexical analysis of social media
• Global Syntactic Variation in Seven Languages: Towards a Computational Dialectology
• Simulating Lexical Semantic Change from Sense-Annotated Data
• Mapping Languages: The Corpus of Global Language Use
• Hahahahaha, Duuuuude, Yeeessss!: A two-parameter characterization of stretchable words and the dynamics of mistypings and misspellings
• Representations of Language Varieties Are Reliable Given Corpus Similarity Measures
• When Dialects Collide: How Socioeconomic Mixing Affects Language Use
• A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains
• Mapping Languages and Demographics with Georeferenced Corpora
• Exploring language relations through syntactic distances and geographic proximity
• The Human Geography of Twitter
• Modeling Global Syntactic Variation in English Using Dialect Classification
• DialectGram: Detecting Dialectal Variation at Multiple Geographic Resolutions