132 citations · 262 across the 12 of their papers we have counts for
14 papers
How Susceptible are LLMs to Influence in Prompts?
Sotiris Anagnostidis, Jannis Bulian
Large Language Models (LLMs) are highly sensitive to prompts, including additional context provided therein. As LLMs grow in capability, understanding their prompt-sensitivity beco…
On scalable oversight with weak LLMs judging strong LLMs
Zachary Kenton, Noah Y. Siegel, János Kramár +8
Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a judge; consultancy, whe…
Assessing Large Language Models on Climate Information
Jannis Bulian, Mike S. Schäfer, Afra Amini +8
As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grou…
Scaling Up Models and Data with and
Adam Roberts, Hyung Won Chung, Anselm Levskaya +40
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can…
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Jannis Bulian, Christian Buck, Wojciech Gajewski +2
The predictions of question answering (QA)systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that re…
Fool Me Twice: Entailment from Wikipedia Gamification
Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bulian +2
We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, dra…
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian +2
We introduce CLIMATE-FEVER, a new publicly available dataset for verification of climate change-related claims. By providing a dataset for the research community, we aim to facilit…
Meta Answering for Machine Reading
Benjamin Borschinger, Jordan Boyd-Graber, Christian Buck +7
We investigate a framework for machine reading, inspired by real world information-seeking problems, where a meta question answering system interacts with a black box environment.…
Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation
Rodrigo Nogueira, Jannis Bulian, Massimiliano Ciaramita
We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the prop…
Analyzing Language Learned by an Active Question Answering Agent
Christian Buck, Jannis Bulian, Massimiliano Ciaramita +4
We analyze the language learned by an agent trained with reinforcement learning as a component of the ActiveQA system [Buck et al., 2017]. In ActiveQA, question answering is framed…
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
Christian Buck, Jannis Bulian, Massimiliano Ciaramita +4
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black bo…
Fixed-parameter Tractable Distances to Sparse Graph Classes
Jannis Bulian, Anuj Dawar
We show that for various classes C of sparse graphs, and several measures of distance to such classes (such as edit distance and elimination distance), the problem of determining t…
Graph Isomorphism Parameterized by Elimination Distance to Bounded Degree
Jannis Bulian, Anuj Dawar
A commonly studied means of parameterizing graph problems is the deletion distance from triviality (Guo et al. 2004), which counts vertices that need to be deleted from a graph to…
Bare canonicity of representable cylindric and polyadic algebras
Jannis Bulian, Ian Hodkinson
We show that for finite n at least 3, every first-order axiomatisation of the varieties of representable n-dimensional cylindric algebras, diagonal-free cylindric algebras, polyadi…