4 papers
HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri +14
We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native…
Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana +10
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually a…
Multicultural Spyfall: Assessing LLMs through a Dynamic Multilingual Social Deduction Game
Haryo Akbarianto Wibowo, Alaa Elsetohy, Qinrong Cui +1
The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturat…
v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
Zhengpeng Shi, Yanpeng Zhao, Jianqun Zhou +6
AI models capable of comprehending humor hold real-world promise -- for example, enhancing engagement in human-machine interactions. To gauge and diagnose the capacity of multimoda…