Publications (61)
Position-aware Self-attention with Relative Positional Encodings for Slot Filling
Ivan Bilan, Benjamin Roth
This paper describes how to apply self-attention with relative positional encodings to the task of relation extraction. We propose to use the self-attention encoder layer together…
MDC-R: The Minecraft Dialogue Corpus with Reference
Chris Madge, Maris Camilleri, Paloma Carretero Garcia +6
We introduce the Minecraft Dialogue Corpus with Reference (MDC-R). MDC-R is a new language resource that supplements the original Minecraft Dialogue Corpus (MDC) with expert annota…
Interpretable Textual Neuron Representations for NLP
Nina Poerner, Benjamin Roth, Hinrich Schütze
Input optimization methods, such as Google Deep Dream, create interpretable representations of neurons for computer vision DNNs. We propose and evaluate ways of transferring this t…
Text-Guided Image Clustering
Andreas Stephan, Lukas Miklautz, Kevin Sidak +4
Image clustering divides a collection of images into meaningful groups, typically interpreted post-hoc via human-given annotations. Those are usually in the form of text, begging t…
Aligning Very Small Parallel Corpora Using Cross-Lingual Word Embeddings and a Monogamy Objective
Nina Poerner, Masoud Jalili Sabet, Benjamin Roth +1
Count-based word alignment methods, such as the IBM models or fast-align, struggle on very small parallel corpora. We therefore present an alternative approach based on cross-lingu…
Cross-functional Analysis of Generalisation in Behavioural Learning
Pedro Henrique Luz de Araujo, Benjamin Roth
In behavioural testing, system functionalities underrepresented in the standard evaluation setting (with a held-out test set) are validated through controlled input-output pairs. O…
Joint Bootstrapping Machines for High Confidence Relation Extraction
Pankaj Gupta, Benjamin Roth, Hinrich Schütze
Semi-supervised bootstrapping techniques for relationship extraction from text iteratively expand a set of initial seed instances. Due to the lack of labeled data, a key challenge…
Evaluating neural network explanation methods using hybrid documents and morphological agreement
Nina Poerner, Benjamin Roth, Hinrich Schütze
The behavior of deep neural networks (DNNs) is hard to understand. This makes it necessary to explore post hoc explanation methods. We conduct the first comprehensive evaluation of…
Counterfactual Reasoning with Knowledge Graph Embeddings
Lena Zellinger, Andreas Stephan, Benjamin Roth
Knowledge graph embeddings (KGEs) were originally developed to infer true but missing facts in incomplete knowledge repositories. In this paper, we link knowledge graph completion…
Compact Example-Based Explanations for Language Models
Loris Schoenegger, Benjamin Roth
Training data influence estimation methods quantify the contribution of training documents to a model's output, making them a promising source of information for example-based expl…
Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations
Pingjun Hong, Beiduo Chen, Siyao Peng +3
Natural Language Inference (NLI) datasets often exhibit human label variation. To better understand these variations, explanation-based approaches analyze the underlying reasoning…
MemeGraphs: Linking Memes to Knowledge Graphs
Vasiliki Kougia, Simon Fetzel, Thomas Kirchmair +4
Memes are a popular form of communicating trends and ideas in social media and on the internet in general, combining the modalities of images and text. They can express humor and s…
Text-Guided Alternative Image Clustering
Andreas Stephan, Lukas Miklautz, Collin Leiber +4
Traditional image clustering techniques only find a single grouping within visual data. In particular, they do not provide a possibility to explicitly define multiple types of clus…
Focused Contrastive Training for Test-based Constituency Analysis
Benjamin Roth, Erion Ãano
We propose a scheme for self-training of grammaticality models for constituency analysis based on linguistic tests. A pre-trained language model is fine-tuned by contrastive estima…
Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions
Pedro Henrique Luz de Araujo, Michael A. Hedderich, Ali Modarressi +2
Persona-assigned large language models (LLMs) are used in domains such as education, healthcare, and sociodemographic simulation. Yet, they are typically evaluated only in short, s…
Interpretable Question Answering on Knowledge Bases and Text
Alona Sydorova, Nina Poerner, Benjamin Roth
Interpretability of machine learning (ML) models becomes more relevant with their increasing adoption. In this work, we address the interpretability of ML based question answering…
Compositional Vector Space Models for Knowledge Base Completion
Arvind Neelakantan, Benjamin Roth, Andrew McCallum
Knowledge base (KB) completion adds new facts to a KB by making inferences from existing facts, for example by inferring with high likelihood nationality(X,Y) from bornIn(X,Y). Mos…
Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency
Vasiliki Kougia, Anastasiia Sedova, Andreas Stephan +2
This paper presents the first study for temporal relation extraction in a zero-shot setting focusing on biomedical text. We employ two types of prompts and five LLMs (GPT-3.5, Mixt…
Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis
Yuxi Xia, Kinga StaÅczak, Benjamin Roth
AI-text detectors achieve high accuracy on in-domain benchmarks, but often struggle to generalize across different generation conditions such as unseen prompts, model families, or…
Proceedings of the First Workshop on Weakly Supervised Learning (WeaSuL)
Michael A. Hedderich, Benjamin Roth, Katharina Kann +3
Welcome to WeaSuL 2021, the First Workshop on Weakly Supervised Learning, co-located with ICLR 2021. In this workshop, we want to advance theory, methods and tools for allowing exp…
Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior
Pedro Henrique Luz de Araujo, Benjamin Roth
One way to personalize and steer generations from large language models (LLM) is to assign a persona: a role that describes how the user expects the LLM to behave (e.g., a helpful…
Black-box Model Ensembling for Textual and Visual Question Answering via Information Fusion
Yuxi Xia, Kilm Zaporojets, Benjamin Roth
A diverse range of large language models (LLMs), e.g., ChatGPT, and visual question answering (VQA) models, e.g., BLIP, have been developed for solving textual and visual question…
WeaNF: Weak Supervision with Normalizing Flows
Andreas Stephan, Benjamin Roth
A popular approach to decrease the need for costly manual annotation of large data sets is weak supervision, which introduces problems of noisy labels, coverage and bias. Methods f…
Specification Overfitting in Artificial Intelligence
Benjamin Roth, Pedro Henrique Luz de Araujo, Yuxi Xia +2
Machine learning (ML) and artificial intelligence (AI) approaches are often criticized for their inherent bias and for their lack of control, accountability, and transparency. Cons…
Neural Architectures for Open-Type Relation Argument Extraction
Benjamin Roth, Costanza Conforti, Nina Poerner +2
In this work, we introduce the task of Open-Type Relation Argument Extraction (ORAE): Given a corpus, a query entity Q and a knowledge base relation (e.g.,"Q authored notable work…
Multilingual Relation Extraction using Compositional Universal Schema
Patrick Verga, David Belanger, Emma Strubell +2
Universal schema builds a knowledge base (KB) of entities and relations by jointly embedding all relation types from input KBs as well as textual patterns expressing relations from…
Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations
Lukas Hinterleitner, Loris Schoenegger, Benjamin Roth
Gradient-based methods for instance-based explanation for large language models (LLMs) are hindered by the immense dimensionality of model gradients. In practice, influence estimat…
ACTC: Active Threshold Calibration for Cold-Start Knowledge Graph Completion
Anastasiia Sedova, Benjamin Roth
Self-supervised knowledge-graph completion (KGC) relies on estimating a scoring model over (entity, relation, entity)-tuples, for example, by embedding an initial knowledge graph.…
Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs
Yuxi Xia, Loris Schoenegger, Benjamin Roth
Large language models (LLMs) can increase users' perceived trust by verbalizing confidence in their outputs. However, prior work has shown that LLMs are often overconfident, making…
Functionality learning through specification instructions
Pedro Henrique Luz de Araujo, Benjamin Roth
Test suites assess natural language processing models' performance on specific functionalities: cases of interest involving model robustness, fairness, or particular linguistic cap…
To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
Anastasiia Sedova, Robert Litschko, Diego Frassinelli +2
One of the major aspects contributing to the striking performance of large language models (LLMs) is the vast amount of factual knowledge accumulated during pre-training. Yet, many…
Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy +1
Expert persona prompting -- assigning roles such as expert in math to language models -- is widely used for task improvement. However, prior work shows mixed results on its effecti…
ReInform: Selecting paths with reinforcement learning for contextualized link prediction
Marina Speranskaya, Sameh Methias, Benjamin Roth
We propose to use reinforcement learning to inform transformer-based contextualized link prediction models by providing paths that are most useful for predicting the correct answer…
Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?
Lukas Thoma, Benjamin Roth
In recent years, deep neural language models have made strong progress in various NLP tasks. This work explores one facet of the question whether state-of-the-art NLP models exhibi…
Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-End Neural Networks
Martin Schmitt, Simon Steinheber, Konrad Schreiber +1
In this work, we propose a new model for aspect-based sentiment analysis. In contrast to previous approaches, we jointly model the detection of aspects and the classification of th…
Comparing Convolutional Neural Networks to Traditional Models for Slot Filling
Heike Adel, Benjamin Roth, Hinrich Schütze
We address relation classification in the context of slot filling, the task of finding and evaluating fillers like "Steve Jobs" for the slot X in "X founded Apple". We propose a co…
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings
Jakob Jungmaier, Nora Kassner, Benjamin Roth
Nowadays, classical count-based word embeddings using positive pointwise mutual information (PPMI) weighted co-occurrence matrices have been widely superseded by machine-learning-b…
Domain adaptation for part-of-speech tagging of noisy user-generated text
Luisa März, Dietrich Trautmann, Benjamin Roth
The performance of a Part-of-speech (POS) tagger is highly dependent on the domain ofthe processed text, and for many domains there is no or only very little training data availabl…
UniSent: Universal Adaptable Sentiment Lexica for 1000+ Languages
Ehsaneddin Asgari, Fabienne Braune, Benjamin Roth +2
In this paper, we introduce UniSent universal sentiment lexica for languages. Sentiment lexica are vital for sentiment analysis in absence of document-level annotations, a…
An Evaluation of Explanation Methods for Black-Box Detectors of Machine-Generated Text
Loris Schoenegger, Yuxi Xia, Benjamin Roth
The increasing difficulty to distinguish language-model-generated from human-written text has led to the development of detectors of machine-generated text (MGT). However, in many…
Data Centric Domain Adaptation for Historical Text with OCR Errors
Luisa März, Stefan Schweter, Nina Poerner +2
We propose new methods for in-domain and cross-domain Named Entity Recognition (NER) on historical data for Dutch and French. For the cross-domain case, we address domain shift by…
KnowMAN: Weakly Supervised Multinomial Adversarial Networks
Luisa März, Ehsaneddin Asgari, Fabienne Braune +2
The absence of labeled data for training neural models is often addressed by leveraging knowledge about the specific task, resulting in heuristic but noisy labels. The knowledge is…
Effective Slot Filling Based on Shallow Distant Supervision Methods
Benjamin Roth, Tassilo Barth, Michael Wiegand +2
Spoken Language Systems at Saarland University (LSV) participated this year with 5 runs at the TAC KBP English slot filling track. Effective algorithms for all parts of the pipelin…
From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
Andreas Stephan, Dawei Zhu, Matthias AÃenmacher +2
To reduce the need for human annotations, large language models (LLMs) have been proposed as judges of the quality of other candidate models. The performance of LLM judges is typic…
Exploring prompts to elicit memorization in masked language model-based named entity recognition
Yuxi Xia, Anastasiia Sedova, Pedro Henrique Luz de Araujo +3
Training data memorization in language models impacts model capability (generalization) and safety (privacy risk). This paper focuses on analyzing prompts' impact on detecting the…
Ranking vs. Classifying: Measuring Knowledge Base Completion Quality
Marina Speranskaya, Martin Schmitt, Benjamin Roth
Knowledge base completion (KBC) methods aim at inferring missing facts from the information present in a knowledge base (KB) by estimating the likelihood of candidate facts. In the…
Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations
Pingjun Hong, Benjamin Roth
Large Language Models (LLMs) can produce verbalized self-explanations, yet prior studies suggest that such rationales may not reliably reflect the model's true decision process. We…
Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles
Yuxi Xia, Pedro Henrique Luz de Araujo, Klim Zaporojets +1
Calibration, the alignment between model confidence and prediction accuracy, is critical for the reliable deployment of large language models (LLMs). Existing works neglect to meas…
ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision
Anastasiia Sedova, Benjamin Roth
A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs), ru…
XPASC: Measuring Generalization in Weak Supervision by Explainability and Association
Luisa März, Ehsaneddin Asgari, Fabienne Braune +2
Weak supervision is leveraged in a wide range of domains and tasks due to its ability to create massive amounts of labeled data, requiring only little manual effort. Standard appro…
Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales
Pingjun Hong, Benjamin Roth
Natural-language explanations are often treated as a unified interface for understanding model behavior, but different explanation sources may support simulation in different ways.…
Checking HateCheck: a cross-functional analysis of behaviour-aware learning for hate speech detection
Pedro Henrique Luz de Araujo, Benjamin Roth
Behavioural testing -- verifying system capabilities by validating human-designed input-output pairs -- is an alternative evaluation method of natural language processing systems p…
Knodle: Modular Weakly Supervised Learning with PyTorch
Anastasiia Sedova, Andreas Stephan, Marina Speranskaya +1
Strategies for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task or integrated with a…
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
Beiduo Chen, Pingjun Hong, Ziyun Zhang +3
Free-text explanations extend human label variation (HLV) beyond label disagreement by revealing the reasoning and preferences behind annotators' decisions. We study whether large…
SepLL: Separating Latent Class Labels from Weak Supervision Noise
Andreas Stephan, Vasiliki Kougia, Benjamin Roth
In the weakly supervised learning paradigm, labeling functions automatically assign heuristic, often noisy, labels to data samples. In this work, we provide a method for learning f…
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Yuxi Xia, Dennis Ulmer, Terra Blevins +3
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the align…
Assessing Wikipedia-Based Cross-Language Retrieval Models
Benjamin Roth
This work compares concept models for cross-language retrieval: First, we adapt probabilistic Latent Semantic Analysis (pLSA) for multilingual documents. Experiments with different…
Influence-driven Curriculum Learning for Pre-training on Limited Data
Loris Schoenegger, Lukas Thoma, Terra Blevins +1
Curriculum learning, a training technique where data is presented to the model in order of example difficulty (e.g., from simpler to more complex documents), has shown limited succ…
Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal
Anastasiia Sedova, Lena Zellinger, Benjamin Roth
An accurate and substantial dataset is essential for training a reliable and well-performing model. However, even manually annotated datasets contain label errors, not to mention a…
Do language models accommodate their users? A study of linguistic convergence
Terra Blevins, Susanne Schmalwieser, Benjamin Roth
While large language models (LLMs) are generally considered proficient in generating language, how similar their language usage is to that of humans remains understudied. In this p…
Topic Segmentation of Research Article Collections
Erion Ãano, Benjamin Roth
Collections of research article data harvested from the web have become common recently since they are important resources for experimenting on tasks such as named entity recogniti…