14 papers
Vision-Language-Guided Pseudo-Labels for Unsupervised Domain Adaptation in Semantic Segmentation for Waste Sorting
Udo Schlegel, Shubhangi, Gabriel Dax +3
Obtaining labeled data for semantic segmentation in applied settings (e.g., autonomous driving, industrial waste sorting) is expensive and often infeasible at scale. We present a c…
TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation
Ruotong Liao, Guowen Huang, Qing Cheng +6
Text-to-video (T2V) generation faces challenging questions when generating videos with long horizons containing multiple events. Inspired by the intrinsics of the diffusion process…
Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs
Jiaqiao Zhang, Zhoujun Li, Zhuojun Li +6
Large reasoning models (LRMs) achieve strong mathematical reasoning performance in English, but remain much less reliable in many low- and medium-resource languages. This gap is of…
NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics
Jian Lan, Zhicheng Liu, Xinpeng Wang +5
The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks suc…
ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations
Yuhao Zhou, Yunpeng Zhu, Yang Zhou +7
Vision-Language-Action (VLA) models hold great promise for general-purpose robotic intelligence, yet scaling up such models is severely bottlenecked by the high cost of acquiring a…
DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
Tanveer Hannan, Dimitrios Mallios, Parth Pathak +5
Large Vision-Language Models (LVLMs) have demonstrated strong multimodal reasoning capabilities on long and complex documents. However, their high memory footprint makes them impra…
PRSM: A Measure to Evaluate CLIP's Robustness Against Paraphrases
Udo Schlegel, Franziska Weeber, Jian Lan +1
Contrastive Language-Image Pre-training (CLIP) is a widely used multimodal model that aligns text and image representations through large-scale training. While it performs strongly…
Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering
Jian Lan, Zhicheng Liu, Udo Schlegel +5
Large vision-language models (VLMs) achieve strong performance in Visual Question Answering but still rely heavily on supervised fine-tuning (SFT) with massive labeled datasets, wh…
SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding
Tanveer Hannan, Shuaicong Wu, Mark Weber +6
A truly capable AI system must do more than detect objects or recognize activities in isolation. It must form unified, grounded representations of who is acting, what they are doin…
Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach
Han Yang, Jian Lan, Yihong Liu +2
Autoregressive language models are vulnerable to orthographic attacks, where input text is perturbed with characters from multilingual alphabets, leading to substantial performance…
AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interaction
Gengyuan Zhang, Tanveer Hannan, Hermine Kleiner +6
An ideal vision-language agent serves as a bridge between the human users and their surrounding physical world in real-world applications like autonomous driving and embodied agent…
Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models
Jian Lan, Udo Schlegel, Tanveer Hannan +3
Although Vision-Language Models (VLMs) have achieved remarkable success, the knowledge mechanisms underlying their social biases remain a black box, where fairness- and ethics-rela…
ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos
Tanveer Hannan, Md Mohaiminul Islam, Jindong Gu +2
Large language models (LLMs) excel at retrieving information from lengthy text, but their vision-language counterparts (VLMs) face difficulties with hour-long videos, especially fo…
RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos
Tanveer Hannan, Md Mohaiminul Islam, Thomas Seidl +1
Locating specific moments within long videos (20-120 minutes) presents a significant challenge, akin to finding a needle in a haystack. Adapting existing short video (5-30 seconds)…