177 citations · 207 across the 16 of their papers we have counts for
17 papers
ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning
İbrahim Ethem Deveci, Funda Tan Çalık, Barış Deniz Sağlam +1
Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathe…
The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
İbrahim Ethem Deveci, Duygu Ataman
The rapid rise of Large Language Models (LLMs) and Large Reasoning Models (LRMs) has been accompanied by an equally rapid increase of benchmarks used to assess them. However, due t…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…
Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluation
Jonne Sälevä, Duygu Ataman, Constantine Lignos
We introduce a set of resampling-based methods for quantifying uncertainty and statistical precision of evaluation metrics in multilingual and/or multitask NLP benchmarks. We show…
TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages
Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili +13
Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However,…
Evaluating Morphological Compositional Generalization in Large Language Models
Mete Ismayilzada, Defne Circi, Jonne Sälevä +6
Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabil…
Generalization Measures for Zero-Shot Cross-Lingual Transfer
Saksham Bassi, Duygu Ataman, Kyunghyun Cho
A model's capacity to generalize its knowledge to interpret unseen inputs with different characteristics is crucial to build robust and reliable machine learning systems. Language…
Logographic Information Aids Learning Better Representations for Natural Language Inference
Zijian Jin, Duygu Ataman
Statistical language models conventionally implement representation learning based on the contextual distribution of words or other formal units, whereas any information related to…
Evaluating Multiway Multilingual NMT in the Turkic Languages
Jamshidbek Mirzakhalov, Anoop Babu, Aigiz Kunafin +11
Despite the increasing number of large and comprehensive machine translation (MT) systems, evaluation of these methods in various languages has been restrained by the lack of high-…
A Large-Scale Study of Machine Translation in the Turkic Languages
Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman +13
Recent advances in neural machine translation (NMT) have pushed the quality of machine translation systems to the point where they are becoming widely adopted to build competitive…
Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models
Jiaoda Li, Duygu Ataman, Rico Sennrich
Multimodal machine translation (MMT) systems have been shown to outperform their text-only neural machine translation (NMT) counterparts when visual context is available. However,…
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Julia Kreutzer, Isaac Caswell, Lisa Wang +49
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text dataset…
On the Importance of Word Boundaries in Character-level Neural Machine Translation
Duygu Ataman, Orhan Firat, Mattia A. Di Gangi +2
Neural Machine Translation (NMT) models generally perform translation using a fixed-size lexical vocabulary, which is an important bottleneck on their generalization capability and…
A Latent Morphology Model for Open-Vocabulary Neural Machine Translation
Duygu Ataman, Wilker Aziz, Alexandra Birch
Translation into morphologically-rich languages challenges neural machine translation (NMT) models with extremely sparse vocabularies where atomic treatment of surface forms is unr…
Bianet: A Parallel News Corpus in Turkish, Kurdish and English
Duygu Ataman
We present a new open-source parallel corpus consisting of news articles collected from the Bianet magazine, an online newspaper that publishes Turkish news, often along with their…
Compositional Representation of Morphologically-Rich Input for Neural Machine Translation
Duygu Ataman, Marcello Federico
Neural machine translation (NMT) models are typically trained with fixed-size input and output vocabularies, which creates an important bottleneck on their accuracy and generalizat…
Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English
Duygu Ataman, Matteo Negri, Marco Turchi +1
The necessity of using a fixed-size word vocabulary in order to control the model complexity in state-of-the-art neural machine translation (NMT) systems is an important bottleneck…