15 citations · 26 across the 7 of their papers we have counts for
9 papers
A Technical Typology of AI Systems in Public Administration
Jonathan Rystrøm, Chris Schmitz, Nathan Davies +3
Research on artificial intelligence (AI) in the public sector often treats "AI" as a single category, neglecting technical distinctions between different AI systems. But these dist…
The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning
Will Hawkins, Kaivalya Rawal, Jonathan Rystrøm +8
Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability…
Grounding Text Embeddings in Stakeholder Associations
Jonathan Rystrøm, Sofie Burgos-Thorsen, Zihao Fu +3
Text embeddings are widely used to analyse large corpora of complex texts. However, it is unclear whether the embeddings capture the same semantic distances as the human experts us…
Agent Benchmarks Fail Public Sector Requirements
Jonathan Rystrøm, Chris Schmitz, Karolina Korgul +2
Deploying Large Language Model-based agents (LLM agents) in the public sector requires assuring that they meet the stringent legal, procedural, and structural requirements of publi…
OxEnsemble: Fair Ensembles for Low-Data Classification
Jonathan Rystrøm, Zihao Fu, Chris Russell
We address the problem of fair classification in settings where data is scarce and unbalanced across demographic groups. Such low-data regimes are common in domains like medical im…
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou +39
Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstrac…
Oversight Structures for Agentic AI in Public-Sector Organizations
Chris Schmitz, Jonathan Rystrøm, Jan Batzner
This paper finds that the introduction of agentic AI systems intensifies existing challenges to traditional public sector oversight mechanisms -- which rely on siloed compliance un…
Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs
Jonathan Rystrøm, Hannah Rose Kirk, Scott Hale
Large Language Models (LLMs) are becoming increasingly capable across global languages. However, the ability to communicate across languages does not necessarily translate to appro…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…