5 citations · 8 across the 23 of their papers we have counts for
25 papers
Rethinking Multi-Image Re-Representation in Multi-Image Understanding
Gengyuan Zhang, Xiao Han, Xinyu Xie +2
Multi-image understanding requires MLLMs not only to recognise the content of individual images, but also to organise visual evidence distributed across them. We study this problem…
QuoteBench: How Matched Scores Can Hide Command-Path Failures
Shangao Li, Yao Zhang, Volker Tresp +1
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation er…
WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents
Yao Zhang, Shijie Tang, Zeyu Li +2
Web agents hold great potential for automating complex computer tasks, yet their interactions involve long-horizon, sequential decision-making with irreversible actions. In such se…
AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models
Haokun Chen, Jianing Li, Yao Zhang +4
Multimodal Large Language Models (MLLMs) achieve impressive performance once optimized on massive datasets. Such datasets often contain sensitive or copyrighted content, raising si…
GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
Yao Zhang, Yu Wu, Haowei Zhang +6
Process Reward Models (PRMs) aim to improve multi-step reasoning in Large Language Models (LLMs) by supervising intermediate steps and identifying errors. However, building effecti…
Deep Research Agents Brings Deeper Harm
Shuo Chen, Zonggen Li, Xingyu Jin +8
We reveal that Deep Research (DR) agents systematically expose safety risks: simply submitting harmful queries that a standalone LLM would reject outright can elicit detailed and d…
Strong but Brittle: Simple Attacks Subvert Reasoning-based Safety Guardrails
Shuo Chen, Zhen Han, Haokun Chen +6
Open-weight Large Reasoning Models (LRMs) are approaching the capabilities of their frontier counterparts but pose significant safety concerns, as they are difficult to patch or mo…
True Multimodal In-Context Learning Needs Attention to the Visual Context
Shuo Chen, Jianzhe Liu, Zhen Han +5
Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks from a few multimodal demon…
Does Machine Unlearning Truly Remove Knowledge?
Haokun Chen, Yueqi Zhang, Yuan Bi +9
In recent years, Large Language Models (LLMs) have achieved remarkable advancements, drawing significant attention from the research community. Their capabilities are largely attri…
Supposedly Equivalent Facts That Aren't? Entity Frequency in Pre-training Induces Asymmetry in LLMs
Yuan He, Bailan He, Zifeng Ding +8
Understanding and mitigating hallucinations in Large Language Models (LLMs) is crucial for ensuring reliable content generation. While previous research has primarily focused on "w…
Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs
Gengyuan Zhang, Mingcong Ding, Tong Liu +2
Multimodal large language models (MLLMs) have demonstrated strong performance in understanding videos holistically, yet their ability to process streaming videos-videos are treated…
FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
Tong Liu, Xiao Yu, Wenxuan Zhou +2
Efficient preference optimization algorithms such as Direct Preference Optimization (DPO) have become a popular approach in aligning large language models (LLMs) with human prefere…
PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
Yilun Liu, Yunpu Ma, Shuo Chen +4
The Mixture-of-Experts (MoE) paradigm has emerged as a powerful approach for scaling transformers with improved resource utilization. However, efficiently fine-tuning MoE models re…
Visual Question Decomposition on Multimodal Large Language Models
Haowei Zhang, Jianzhe Liu, Zhen Han +5
Question decomposition has emerged as an effective strategy for prompting Large Language Models (LLMs) to answer complex questions. However, while existing methods primarily focus…
VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs
Ruotong Liao, Max Erler, Huiyu Wang +4
In the video-language domain, recent works in leveraging zero-shot Large Language Model-based reasoning for video understanding have become competitive challengers to previous end-…
FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models
Haokun Chen, Hang Li, Yao Zhang +7
One-Shot Federated Learning (OSFL), a special decentralized machine learning paradigm, has recently gained significant attention. OSFL requires only a single round of client data o…
Multimodal Pragmatic Jailbreak on Text-to-image Models
Tong Liu, Zhixin Lai, Jiawen Wang +6
Diffusion models have recently achieved remarkable advancements in terms of image quality and fidelity to textual prompts. Concurrently, the safety of such generative models has be…
How the (Tensor-) Brain uses Embeddings and Embodiment to Encode Senses and Symbols
Volker Tresp, Hang Li
The Tensor Brain (TB) has been introduced as a computational model for perception and memory. This paper provides an overview of the TB model, incorporating recent developments and…
Provably Better Explanations with Optimized Aggregation of Feature Attributions
Thomas Decker, Ananta R. Bhattarai, Jindong Gu +2
Using feature attributions for post-hoc explanations is a common practice to understand and verify the predictions of opaque machine learning models. Despite the numerous technique…
Localizing Events in Videos with Multimodal Queries
Gengyuan Zhang, Mang Ling Ada Fok, Jialu Ma +5
Localizing events in videos based on semantic queries is a pivotal task in video understanding, with the growing significance of user-oriented applications like video search. Yet,…
Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?
Shuo Chen, Zhen Han, Bailan He +5
Various jailbreak attacks have been proposed to red-team Large Language Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some methods are not limited to the t…
Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image
Zefeng Wang, Zhen Han, Shuo Chen +6
Multimodal LLMs (MLLMs) with a great ability of text and image understanding have received great attention. To achieve better reasoning with MLLMs, Chain-of-Thought (CoT) reasoning…
Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining
Ugur Sahin, Hang Li, Qadeer Khan +2
Contemporary large-scale visual language models (VLMs) exhibit strong representation capacities, making them ubiquitous for enhancing image and text understanding tasks. They are o…
Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
Shuo Chen, Zhen Han, Bailan He +6
Large Language Models (LLMs) with in-context learning (ICL) ability can quickly adapt to a specific context given a few demonstrations (demos). Recently, Multimodal Large Language…