15 citations · 22 across the 20 of their papers we have counts for
23 papers
Multilingual GSM-Symbolic: What determines capability transfer across languages?
Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard +22
We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datase…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Mimir: A Sub-2B Hierarchical Reasoning Model Trained on Permissible Data Punches Above Its Weight Class
Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina +4
Training capable language models typically relies on web-scale pretraining followed by multi-stage post-training, creating substantial computational and data-governance barriers fo…
MVEB: Massive Video Embedding Benchmark
Adnan El Assadi, Roman Solomatin, Isaac Chung +13
We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classificati…
Modeling semantic association in self-paced reading with language model embeddings
Sara Møller Østergaard, Kenneth Enevoldsen, Afra Alishahi +1
Semantic association between a word and its context has been identified as an important component of reading comprehension, even when word predictability is accounted for. Recent r…
Grounding Text Embeddings in Stakeholder Associations
Jonathan Rystrøm, Sofie Burgos-Thorsen, Zihao Fu +3
Text embeddings are widely used to analyse large corpora of complex texts. However, it is unclear whether the embeddings capture the same semantic distances as the human experts us…
Naturalistic measure of social norms alignment
Yevhen Kostiuk, Kenneth Enevoldsen, Peter Bjerregaard Vahlstrup +2
Social norms reflect shared expectations on acceptable behavior. Measuring social norms alignment remains challenging, with existing approaches typically relying on artificial clos…
Closed-form predictive coding via hierarchical Gaussian filters
Aleksandrs Baskakovs, Sylvain Estebe, Kenneth Enevoldsen +3
Predictive coding (PC) offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks, yet to date, it remains slower, and pe…
One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
Yevhen Kostiuk, Kenneth Enevoldsen
Instruction embedding models have become common among state-of-the-art models, however are evaluated using a single prompt per task. The single-point evaluation ignores a main prob…
Improving reasoning at inference time via uncertainty minimisation
Nicolas Legrand, Kenneth Enevoldsen, Márton Kardos +1
Large language models (LLMs) now exhibit strong multi-step reasoning abilities, but existing inference-time scaling methods remain computationally expensive, often relying on exten…
MAEB: Massive Audio Embedding Benchmark
Adnan El Assadi, Isaac Chung, Chenghao Xiao +15
We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasonin…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…
HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks
Adnan El Assadi, Isaac Chung, Roman Solomatin +2
Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where t…
Dynaword: From One-shot to Continuously Developed Datasets
Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan +14
Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguousl…
Continuous sentiment scores for literary and multilingual contexts
Laurits Lyngbaek, Pascale Feldkamp, Yuri Bizzoni +2
Sentiment Analysis is widely used to quantify sentiment in text, but its application to literary texts poses unique challenges due to figurative language, stylistic ambiguity, as w…
Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks
Isaac Chung, Imene Kerboua, Marton Kardos +2
The Massive Text Embedding Benchmark (MTEB) has become a standard evaluation platform for text embedding models. While previous work has established the core benchmark methodology,…
topicwizard -- a Modern, Model-agnostic Framework for Topic Model Visualization and Interpretation
Márton Kardos, Kenneth C. Enevoldsen, Kristoffer Laigaard Nielbo
Topic models are statistical tools that allow their users to gain qualitative and quantitative insights into the contents of textual corpora without the need for close reading. The…
MIEB: Massive Image Embedding Benchmark
Chenghao Xiao, Isaac Chung, Imene Kerboua +7
Image representations are often evaluated through disjointed, task-specific protocols, leading to a fragmented understanding of model capabilities. For instance, it is unclear whet…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…
The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff +1
The evaluation of English text embeddings has transitioned from evaluating a handful of datasets to broad coverage across many tasks through benchmarks such as MTEB. However, this…
Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks
Dan Saattrup Nielsen, Kenneth Enevoldsen, Peter Schneider-Kamp
This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Bui…
-- Semantic Signal Separation
Márton Kardos, Jan Kostkan, Arnault-Quentin Vermillet +3
Topic models are useful tools for discovering latent semantic structures in large textual corpora. Recent efforts have been oriented at incorporating contextual representations in…
Augmenty: A Python Library for Structured Text Augmentation
Kenneth Enevoldsen
Augmnety is a Python library for structured text augmentation. It is built on top of spaCy and allows for augmentation of both the text and its annotations. Augmenty provides a wid…