2 citations · 2 across the 11 of their papers we have counts for
12 papers
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
David Schlangen, Sherzod Hakimov, Chalamalasetti Kranti +2
There are currently two main paradigms for evaluating large language models (LLMs), reference-based evaluation and preference-based evaluation. The first, carried over from the eva…
Playpen: An Environment for Exploring Learning Through Conversational Interaction
Nicola Horst, Davide Mazzaccara, Antonia Schmidt +13
Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appro…
The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization
Luka Borec, Philipp Sadler, David Schlangen
This work analyses the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling. Stochastic decoding methods like nucleus sampling are typicall…
clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
Anne Beyer, Kranti Chalamalasetti, Sherzod Hakimov +3
It has been established in recent work that Large Language Models (LLMs) can be prompted to "self-play" conversational games that probe certain capabilities (general instruction fo…
Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies
Philipp Sadler, Sherzod Hakimov, David Schlangen
In collaborative goal-oriented settings, the participants are not only interested in achieving a successful outcome, but do also implicitly negotiate the effort they put into the i…
Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Game
Philipp Sadler, Sherzod Hakimov, David Schlangen
Albrecht and Stone (2018) state that modeling of changing behaviors remains an open problem "due to the essentially unconstrained nature of what other agents may do". In this work…
Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples
Philipp Sadler, David Schlangen
NLP tasks are typically defined extensionally through datasets containing example instantiations (e.g., pairs of image i and text t), but motivated intensionally through capabiliti…
Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers
Philipp Sadler, Sherzod Hakimov, David Schlangen
The ability to pick up on language signals in an ongoing interaction is crucial for future machine learning models to collaborate and interact with humans naturally. In this paper,…
Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents
Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov +3
Recent work has proposed a methodology for the systematic evaluation of "Situated Language Understanding Agents"-agents that operate in rich linguistic and non-linguistic contexts-…
Spatial Attention as an Interface for Image Captioning Models
Philipp Sadler
The internal workings of modern deep learning models stay often unclear to an external observer, although spatial attention mechanisms are involved. The idea of this work is to tra…
Can Neural Image Captioning be Controlled via Forced Attention?
Philipp Sadler, Tatjana Scheffler, David Schlangen
Learned dynamic weighting of the conditioning signal (attention) has been shown to improve neural language generation in a variety of settings. The weights applied when generating…
Detecting cities in aerial night-time images by learning structural invariants using single reference augmentation
Philipp Sadler
This paper examines, if it is possible to learn structural invariants of city images by using only a single reference picture when producing transformations along the variants in t…