37 citations · 79 across the 20 of their papers we have counts for
25 papers
GRAFITE: Generative Regression Analysis Framework for Issue Tracking and Evaluation
Ja Young Lee, Mírian Silva, Mohamed Nasr +6
Large language models (LLMs) are largely motivated by their performance on popular topics and benchmarks at the time of their release. However, over time, contamination occurs due…
MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
Sara Rosenthal, Yannis Katsis, Vraj Shah +3
We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 t…
A Longitudinal Study on Different Annotator Feedback Loops in Complex RAG Tasks
Sara Rosenthal, Maeda Hanafi, Yannis Katsis +2
Grounding conversations in existing passages, known as Retrieval-Augmented Generation (RAG), is an important aspect of Chat-Based Assistants powered by Large Language Models (LLMs)…
RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits
Kshitij Fadnis, Sara Rosenthal, Maeda Hanafi +2
Retrieval Augmented Generation (RAG) is an important aspect of conversing with Large Language Models (LLMs) when factually correct information is important. LLMs may provide answer…
Granite Embedding Models
Parul Awasthy, Aashka Trivedi, Yulong Li +19
We introduce the Granite Embedding models, a family of encoder-based embedding models designed for retrieval tasks, spanning dense-retrieval and sparse retrieval architectures, wit…
MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Yannis Katsis, Sara Rosenthal, Kshitij Fadnis +7
Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is…
InspectorRAGet: An Introspection Platform for RAG Evaluation
Kshitij Fadnis, Siva Sankalp Patel, Odellia Boni +4
Large Language Models (LLM) have become a popular approach for implementing Retrieval Augmented Generation (RAG) systems, and a significant amount of effort has been spent on build…
CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems
Sara Rosenthal, Avirup Sil, Radu Florian +1
Retrieval Augmented Generation (RAG) has become a popular application for large language models. It is preferable that successful RAG systems provide accurate answers that are supp…
Muted: Multilingual Targeted Offensive Speech Identification and Visualization
Christoph Tillmann, Aashka Trivedi, Sara Rosenthal +4
Offensive language such as hate, abuse, and profanity (HAP) occurs in various content on the web. While previous work has mostly dealt with sentence level annotations, there have b…
PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development
Avirup Sil, Jaydeep Sen, Bhavani Iyer +12
The field of Question Answering (QA) has made remarkable progress in recent years, thanks to the advent of large pre-trained language models, newer realistic benchmark datasets wit…
GAAMA 2.0: An Integrated System that Answers Boolean and Extractive Questions
Scott McCarley, Mihaela Bornea, Sara Rosenthal +4
Recent machine reading comprehension datasets include extractive and boolean questions but current approaches do not offer integrated support for answering both question types. We…
Task Transfer and Domain Adaptation for Zero-Shot Question Answering
Xiang Pan, Alex Sheng, David Shimshoni +3
Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning method…
Do Answers to Boolean Questions Need Explanations? Yes
Sara Rosenthal, Mihaela Bornea, Avirup Sil +2
Existing datasets that contain boolean questions, such as BoolQ and TYDI QA , provide the user with a YES/NO response to the question. However, a one word response is not sufficien…
SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)
Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky +1
Understanding tables is an important and relevant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this p…
Are Multilingual BERT models robust? A Case Study on Adversarial Attacks for Multilingual Question Answering
Sara Rosenthal, Mihaela Bornea, Avirup Sil
Recent approaches have exploited weaknesses in monolingual question answering (QA) models by adding adversarial statements to the passage. These attacks caused a reduction in state…
Multilingual Transfer Learning for QA Using Translation as Data Augmentation
Mihaela Bornea, Lin Pan, Sara Rosenthal +2
Prior work on multilingual question answering has mostly focused on using large multilingual pre-trained language models (LM) to perform zero-shot language-wise learning: train a Q…
SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)
Marcos Zampieri, Preslav Nakov, Sara Rosenthal +6
We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtas…
SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification
Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov +2
The widespread use of offensive content in social media has led to an abundance of research in detecting language such as hate speech, cyberbullying, and cyber-aggression. Recent w…
SemEval-2013 Task 2: Sentiment Analysis in Twitter
Preslav Nakov, Zornitsa Kozareva, Alan Ritter +3
In recent years, sentiment analysis in social media has attracted a lot of research interest and has been used for a number of applications. Unfortunately, research has been hinder…
SemEval-2014 Task 9: Sentiment Analysis in Twitter
Sara Rosenthal, Preslav Nakov, Alan Ritter +1
We describe the Sentiment Analysis in Twitter task, ran as part of SemEval-2014. It is a continuation of the last year's task that ran successfully as part of SemEval-2013. As in 2…
SemEval-2015 Task 10: Sentiment Analysis in Twitter
Sara Rosenthal, Saif M Mohammad, Preslav Nakov +3
In this paper, we describe the 2015 iteration of the SemEval shared task on Sentiment Analysis in Twitter. This was the most popular sentiment analysis shared task to date with mor…
SemEval-2016 Task 4: Sentiment Analysis in Twitter
Preslav Nakov, Alan Ritter, Sara Rosenthal +2
This paper discusses the fourth year of the ``Sentiment Analysis in Twitter Task''. SemEval-2016 Task 4 comprises five subtasks, three of which represent a significant departure fr…
SemEval-2017 Task 4: Sentiment Analysis in Twitter
Sara Rosenthal, Noura Farra, Preslav Nakov
This paper describes the fifth year of the Sentiment Analysis in Twitter task. SemEval-2017 Task 4 continues with a rerun of the subtasks of SemEval-2016 Task 4, which include iden…
SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
Marcos Zampieri, Shervin Malmasi, Preslav Nakov +3
We present the results and the main findings of SemEval-2019 Task 6 on Identifying and Categorizing Offensive Language in Social Media (OffensEval). The task was based on a new dat…