70 citations · 145 across the 19 of their papers we have counts for
22 papers
Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Models
Athina Kyriakou, Dennis Ulmer, Ivan Titov
Confidence estimation (CE), i.e. quantifying the reliability of a model's prediction, has attracted great interest in the context of large language models (LLMs). However, most stu…
Probing for Knowledge Attribution in Large Language Models
Ivo Brink, Alexander Boer, Dennis Ulmer
Large language model (LLM) hallucinations, meaning fluent but factually incorrect generations, fall into two types: faithfulness violations, where the model misuses provided contex…
Queer NLP: A Critical Survey on Literature Gaps, Biases and Trends
Sabine Weber, Angelina Wang, Ankush Gupta +16
Natural language processing (NLP) technologies are rapidly reshaping how language is created, processed, and interpreted by humans. With current and potential applications in hirin…
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Yuxi Xia, Dennis Ulmer, Terra Blevins +3
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the align…
Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
Dennis Ulmer, Alexandra Lorson, Ivan Titov +1
Human users increasingly communicate with large language models (LLMs), but LLMs suffer from frequent overconfidence in their output, even when its accuracy is questionable, which…
On Uncertainty In Natural Language Processing
Dennis Ulmer
The last decade in deep learning has brought on increasingly capable systems that are deployed on a wide variety of applications. In natural language processing, the field has been…
Calibrating Large Language Models Using Their Generations Only
Dennis Ulmer, Martin Gubri, Hwaran Lee +2
As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model's confidence in its p…
TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification
Martin Gubri, Dennis Ulmer, Hwaran Lee +2
Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. Assessing the compliance of the released LLMs is crucial,…
Non-Exchangeable Conformal Language Generation with Nearest Neighbors
Dennis Ulmer, Chrysoula Zerva, André F. T. Martins
Quantifying uncertainty in automatically generated text is important for letting humans check potential hallucinations and making systems more reliable. Conformal prediction is an…
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
Dennis Ulmer, Elman Mansimov, Kaixiang Lin +3
Large language models (LLMs) are powerful dialogue agents, but specializing them towards fulfilling a specific function can be challenging. Instructing tuning, i.e. tuning models o…
Non-Exchangeable Conformal Risk Control
António Farinhas, Chrysoula Zerva, Dennis Ulmer +1
Split conformal prediction has recently sparked great interest due to its ability to provide formally guaranteed uncertainty sets or intervals for predictions made by black-box neu…
Uncertainty in Natural Language Generation: From Theory to Applications
Joris Baan, Nico Daheim, Evgenia Ilia +7
Recent advances of powerful Language Models have allowed Natural Language Generation (NLG) to emerge as an important technology that can not only perform traditional tasks like sum…
Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity
Dennis Ulmer, Jes Frellsen, Christian Hardmeier
We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training mod…
State-of-the-art generalisation research in NLP: A taxonomy and review
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers +17
The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not…
deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks
Dennis Ulmer, Christian Hardmeier, Jes Frellsen
A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endang…
Experimental Standards for Deep Learning in Natural Language Processing Research
Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein +5
The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more…
Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
Dennis Ulmer, Christian Hardmeier, Jes Frellsen
Popular approaches for quantifying predictive uncertainty in deep neural networks often involve distributions over weights or multiple models, for instance via Markov Chain samplin…
Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs
Dennis Ulmer
In Recurrent Neural Networks (RNNs), encoding information in a suboptimal or erroneous way can impact the quality of representations based on later elements in the sequence and sub…
Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
Dennis Ulmer, Giovanni Cinà
A crucial requirement for reliable deployment of deep learning models for safety-critical applications is the ability to identify out-of-distribution (OOD) data points, samples whi…
Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data
Dennis Ulmer, Lotta Meijerink, Giovanni Cinà
When deploying machine learning models in high-stakes real-world environments such as health care, it is crucial to accurately assess the uncertainty concerning a model's predictio…
Assessing incrementality in sequence-to-sequence models
Dennis Ulmer, Dieuwke Hupkes, Elia Bruni
Since their inception, encoder-decoder models have successfully been applied to a wide array of problems in computational linguistics. The most recent successes are predominantly d…
On the Realization of Compositionality in Neural Networks
Joris Baan, Jana Leible, Mitja Nikolaus +5
We present a detailed comparison of two types of sequence to sequence models trained to conduct a compositional task. The models are architecturally identical at inference time, bu…