Computational Hermeneutics: Evaluating generative AI as a cultural technology

Abstract

Generative AI systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat culture as a variable to be measured rather than fundamental to the system's operation. Drawing on hermeneutic theory from the humanities, we argue that GenAI systems function as "context machines" that must inherently address three interpretive challenges: situatedness (meaning only emerges in context), plurality (multiple valid interpretations coexist), and ambiguity (interpretations naturally conflict). We present computational hermeneutics as an emerging framework offering an interpretive account of what GenAI systems do, and how they might do it better. We offer three principles for hermeneutic evaluation -- that benchmarks should be iterative, not one-off; include people, not just machines; and measure cultural context, not just model output. This perspective offers a nascent paradigm for designing and evaluating contemporary AI systems: shifting from standardized questions about accuracy to contextual ones about meaning.

Published in Frontiers in Artificial Intelligence

References in corpus (11)

• From Frequency to Meaning: Vector Space Models of Semantics
• The Geometry of Culture: Analyzing Meaning through Word Embeddings
• Unleashing the potential of prompt engineering for large language models
• Cultural Bias and Cultural Alignment of Large Language Models
• The Metacognitive Demands and Opportunities of Generative AI
• Machine Culture
• Cultural Cartography with Word Embeddings
• How Culture Shapes What People Want From AI
• Mapping global dynamics of benchmark creation and saturation in artificial intelligence
• Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals
• Assessing and Understanding Creativity in Large Language Models