11 citations · 22 across the 17 of their papers we have counts for
21 papers
Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing
Raoyuan Zhao, Abdullatif Köksal, Ali Modarressi +2
The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within L…
TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages
Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili +13
Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However,…
How far can bias go? Tracing bias from pretraining data to alignment
Marion Thaler, Abdullatif Köksal, Alina Leidinger +2
As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mi…
Evaluating Morphological Compositional Generalization in Large Language Models
Mete Ismayilzada, Defne Circi, Jonne Sälevä +6
Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabil…
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions
Abdullatif Köksal, Marion Thaler, Ayyoob Imani +3
Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets…
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
Ingo Ziegler, Abdullatif Köksal, Desmond Elliott +1
Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge. We propose Corpus Retrieva…
SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists
Raoyuan Zhao, Abdullatif Köksal, Yihong Liu +3
Traditional benchmarking in NLP typically involves using static held-out test sets. However, this approach often results in an overestimation of performance and lacks the ability t…
TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish
Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel +2
Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automat…
Consistent Document-Level Relation Extraction via Counterfactuals
Ali Modarressi, Abdullatif Köksal, Hinrich Schütze
Many datasets have been developed to train and evaluate document-level relation extraction (RE) models. Most of these are constructed using real-world data. It has been shown that…
MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory
Ali Modarressi, Abdullatif Köksal, Ayyoob Imani +2
While current large language models (LLMs) perform well on many knowledge-related tasks, they are limited by relying on their parameters as an implicit storage mechanism. As a resu…
Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena
Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze
Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (…
Sociocultural knowledge is needed for selection of shots in hate speech detection tasks
Antonis Maronikolakis, Abdullatif Köksal, Hinrich Schütze
We introduce HATELEXICON, a lexicon of slurs and targets of hate speech for the countries of Brazil, Germany, India and Kenya, to aid training and interpretability of models. We de…
Language-Agnostic Bias Detection in Language Models with Bias Probing
Abdullatif Köksal, Omer Faruk Yalcin, Ahmet Akbiyik +3
Pretrained language models (PLMs) are key components in NLP, but they contain strong social biases. Quantifying these biases is challenging because current methods focusing on fill…
LongForm: Effective Instruction Tuning with Reverse Instructions
Abdullatif Köksal, Timo Schick, Anna Korhonen +1
Instruction tuning enables language models to more effectively generalize and better follow user intent. However, obtaining instruction data is costly and challenging. Prior work e…
MEAL: Stable and Active Learning for Few-Shot Prompting
Abdullatif Köksal, Timo Schick, Hinrich Schütze
Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high…
The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative
Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal +1
Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it pos…
SilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignment
Abdullatif Köksal, Silvia Severini, Hinrich Schütze
Word alignments are essential for a variety of NLP tasks. Therefore, choosing the best approaches for their creation is crucial. However, the scarce availability of gold evaluation…
Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution
Yi Huang, Buse Giledereli, Abdullatif Köksal +2
Multi-label text classification is a challenging task because it requires capturing label dependencies. It becomes even more challenging when class distribution is long-tailed. Res…
The RELX Dataset and Matching the Multilingual Blanks for Cross-Lingual Relation Classification
Abdullatif Köksal, Arzucan Özgür
Relation classification is one of the key topics in information extraction, which can be used to construct knowledge bases or to provide useful information for question answering.…
Vapur: A Search Engine to Find Related Protein-Compound Pairs in COVID-19 Literature
Abdullatif Köksal, Hilal Dönmez, Rıza Özçelik +2
Coronavirus Disease of 2019 (COVID-19) created dire consequences globally and triggered an intense scientific effort from different domains. The resulting publications created a hu…
Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Tool
Utku Türk, Furkan Atmaca, Şaziye Betül Özateş +6
In this paper, we introduce the resources that we developed for Turkish dependency parsing, which include a novel manually annotated treebank (BOUN Treebank), along with the guidel…