86 citations · 305 across the 37 of their papers we have counts for
44 papers
Data Contamination Report from the 2024 CONDA Shared Task
Oscar Sainz, Iker García-Ferrero, Alon Jacovi +25
The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as…
Memorization in In-Context Learning
Shahriar Golchin, Mihai Surdeanu, Steven Bethard +2
In-context learning (ICL) has proven to be an effective strategy for improving the performance of large language models (LLMs) with no additional training. However, the exact mecha…
Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation
Fahmida Alam, Md Asiful Islam, Robert Vacareanu +1
We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets NYT29 (Takanobu et al., 2019…
From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu +1
We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any addit…
Best of Both Worlds: A Pliable and Generalizable Neuro-Symbolic Approach for Relation Classification
Robert Vacareanu, Fahmida Alam, Md Asiful Islam +2
This paper introduces a novel neuro-symbolic architecture for relation classification (RC) that combines rule-based methods with contemporary deep learning techniques. This approac…
Enhancing Transformer RNNs with Multiple Temporal Perspectives
Razvan-Gabriel Dumitru, Darius Peteleaza, Mihai Surdeanu
We introduce the concept of multiple temporal perspectives, a novel approach applicable to Recurrent Neural Network (RNN) architectures for enhancing their understanding of sequent…
Divide & Conquer for Entailment-aware Multi-hop Evidence Retrieval
Fan Luo, Mihai Surdeanu
Lexical and semantic matches are commonly used as relevance measurements for information retrieval. Together they estimate the semantic equivalence between the query and the candid…
Perturbation-based Active Learning for Question Answering
Fan Luo, Mihai Surdeanu
Building a question answering (QA) model with less annotation costs can be achieved by utilizing active learning (AL) training strategy. It selects the most informative unlabeled t…
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Shahriar Golchin, Mihai Surdeanu
We propose the Data Contamination Quiz (DCQ), a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifical…
Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Shahriar Golchin, Mihai Surdeanu
Data contamination, i.e., the presence of test data from downstream tasks in the training data of large language models (LLMs), is a potential major issue in measuring LLMs' real e…
Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords
Shahriar Golchin, Mihai Surdeanu, Nazgol Tavabi +1
We propose a novel task-agnostic in-domain pre-training method that sits between generic pre-training and fine-tuning. Our approach selectively masks in-domain keywords, i.e., word…
It is not Sexually Suggestive, It is Educative. Separating Sex Education from Suggestive Content on TikTok Videos
Enfa George, Mihai Surdeanu
We introduce SexTok, a multi-modal dataset composed of TikTok videos labeled as sexually suggestive (from the annotator's point of view), sex-educational content, or neither. Such…
Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference
Sushma Anand Akoju, Robert Vacareanu, Haris Riaz +2
We introduce a synthetic dataset called Sentences Involving Complex Compositional Knowledge (SICCK) and a novel analysis that investigates the performance of Natural Language Infer…
Explainable Verbal Reasoner Plus (EVR+): A Natural Language Reasoning Framework that Supports Diverse Compositional Reasoning
Zhengzhong Liang, Zeyu Zhang, Steven Bethard +1
Languages models have been successfully applied to a variety of reasoning tasks in NLP, yet the language models still suffer from compositional generalization. In this paper we pre…
Validity Assessment of Legal Will Statements as Natural Language Inference
Alice Saebom Kwak, Jacob O. Israelsen, Clayton T. Morrison +2
This work introduces a natural language inference (NLI) dataset that focuses on the validity of statements in legal wills. This dataset is unique because: (a) each entailment decis…
BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples
Mohaddeseh Bastan, Mihai Surdeanu, Niranjan Balasubramanian
Natural language inference (NLI) is critical for complex decision-making in biomedical domain. One key question, for example, is whether a given biomedical mechanism is supported b…
A Compact Pretraining Approach for Neural Language Models
Shahriar Golchin, Mihai Surdeanu, Nazgol Tavabi +1
Domain adaptation for large neural language models (NLMs) is coupled with massive amounts of unstructured data in the pretraining phase. In this study, however, we show that pretra…
Learning Open Domain Multi-hop Search Using Reinforcement Learning
Enrique Noriega-Atala, Mihai Surdeanu, Clayton T. Morrison
We propose a method to teach an automated agent to learn how to search for multi-hop paths of relations between entities in an open domain. The method learns a policy for directing…
SuMe: A Dataset Towards Summarizing Biomedical Mechanisms
Mohaddeseh Bastan, Nishant Shankar, Mihai Surdeanu +1
Can language models read biomedical texts and explain the biomedical mechanisms discussed? In this work we introduce a biomedical mechanism summarization task. Biomedical studies o…
Better Retrieval May Not Lead to Better Question Answering
Zhengzhong Liang, Tushar Khot, Steven Bethard +2
Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC). A popular app…
It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers
Zheng Tang, Mihai Surdeanu
We propose an explainable approach for relation extraction that mitigates the tension between generalization and explainability by jointly training for the two goals. Our approach…
From Examples to Rules: Neural Guided Rule Synthesis for Information Extraction
Robert Vacareanu, Marco A. Valenzuela-Escarcega, George C. G. Barbosa +2
While deep learning approaches to information extraction have had many successes, they can be difficult to augment or maintain as needs shift. Rule-based methods, on the other hand…
Automatic Correction of Syntactic Dependency Annotation Differences
Andrew Zupon, Andrew Carnie, Michael Hammond +1
Annotation inconsistencies between data sets can cause problems for low-resource NLP, where noisy or inconsistent data cannot be as easily replaced compared with resource-rich lang…
Informal Persian Universal Dependency Treebank
Roya Kabiri, Simin Karimi, Mihai Surdeanu
This paper presents the phonological, morphological, and syntactic distinctions between formal and informal Persian, showing that these two variants have fundamental differences th…