Evaluation of Geographical Distortions in Language Models

Abstract

Language models now constitute essential tools for improving efficiency for many professional tasks such as writing, coding, or learning. For this reason, it is imperative to identify inherent biases. In the field of Natural Language Processing, five sources of bias are well-identified: data, annotation, representation, models, and research design. This study focuses on biases related to geographical knowledge. We explore the connection between geography and language models by highlighting their tendency to misrepresent spatial information, thus leading to distortions in the representation of geographical distances. This study introduces four indicators to assess these distortions, by comparing geographical and semantic distances. Experiments are conducted from these four indicators with ten widely used language models. Results underscore the critical necessity of inspecting and rectifying spatial biases in language models to ensure accurate and equitable representations.

Accepted version. Published in Machine Learning (Springer) 114:263 (2025). Open access under a CC BY-NC-ND 4.0 license. DOI: 10.1007/s10994-025-06916-9

References in corpus (10)

• Llama 2: Open Foundation and Fine-Tuned Chat Models
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Geo-knowledge-guided GPT models improve the extraction of location descriptions from disaster-related social media messages
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
• Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
• Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
• SoK: Memorization in General-Purpose Large Language Models
• Leave no Place Behind: Improved Geolocation in Humanitarian Documents
• Quantifying Geospatial in the Common Crawl Corpus