3 citations · 4 across the 9 of their papers we have counts for
9 papers
Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Models
Mohamad Ballout, Serwan Jassim, Elia Bruni
This paper presents a systematic evaluation of state-of-the-art multimodal large language models (MLLMs) on intuitive physics tasks using the GRASP and IntPhys 2 datasets. We asses…
iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs
Julius Mayer, Mohamad Ballout, Serwan Jassim +2
Vision-Language Models (VLMs) are known to struggle with spatial reasoning and visual alignment. To help overcome these limitations, we introduce iVISPAR, an interactive multimodal…
A Review of Nine Physics Engines for Reinforcement Learning Research
Michael Kaup, Cornelius Wolff, Hyerim Hwang +2
We present a review of popular simulation engines and frameworks used in reinforcement learning (RL) research, aiming to guide researchers in selecting tools for creating simulated…
Bidirectional Emergent Language in Situated Environments
Cornelius Wolff, Julius Mayer, Elia Bruni +1
Emergent language research has made significant progress in recent years, but still largely fails to explore how communication emerges in more complex and situated multi-agent syst…
Interpretability of Language Models via Task Spaces
Lucas Weber, Jaap Jumelet, Elia Bruni +1
The usual way to interpret language models (LMs) is to test their performance on different benchmarks and subsequently infer their internal processes. In this paper, we present an…
From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
Xenia Ohmer, Elia Bruni, Dieuwke Hupkes
The staggering pace with which the capabilities of large language models (LLMs) are increasing, as measured by a range of commonly used natural language understanding (NLU) benchma…
The ICL Consistency Test
Lucas Weber, Elia Bruni, Dieuwke Hupkes
Just like the previous generation of task-tuned models, large language models (LLMs) that are adapted to tasks via prompt-based methods like in-context-learning (ICL) perform well…
GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models
Serwan Jassim, Mario Holubar, Annika Richter +3
This paper presents GRASP, a novel benchmark to evaluate the language grounding and physical understanding capabilities of video-based multimodal large language models (LLMs). This…
Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning
Lucas Weber, Elia Bruni, Dieuwke Hupkes
Finding the best way of adapting pre-trained language models to a task is a big challenge in current NLP. Just like the previous generation of task-tuned models (TT), models that a…