129 citations · 494 across the 76 of their papers we have counts for
22 papers · 1 filter
METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding
Mengyue Wang, Shuo Chen, Kristian Kersting +2
Recent advances in Video Large Language Models (VLLMs) have significantly enhanced their ability to understand video content. Nonetheless, processing long videos remains challengin…
PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection
Jinhe Bi, Aniri, Zengjie Jin +11
Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these d…
LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
Jinhe Bi, Yujun Wang, Haokun Chen +4
Multimodal Large Language Models (MLLMs) have significantly advanced visual tasks by integrating visual representations into large language models (LLMs). The textual modality, inh…
SPOT! Revisiting Video-Language Models for Event Understanding
Gengyuan Zhang, Jinhe Bi, Jindong Gu +2
Understanding videos is an important research topic for multimodal learning. Leveraging large-scale datasets of web-crawled video-text pairs as weak supervision has become a pre-tr…
Multi-event Video-Text Retrieval
Gengyuan Zhang, Jisen Ren, Jindong Gu +1
Video-Text Retrieval (VTR) is a crucial multi-modal task in an era of massive video-text data on the Internet. A plethora of work characterized by using a two-stream Vision-Languag…
Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning
Gengyuan Zhang, Yurui Zhang, Kerui Zhang +1
Vision-Language Models (VLMs) are expected to be capable of reasoning with commonsense knowledge as human beings. One example is that humans can reason where and when an image is t…
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
Shuo Chen, Jindong Gu, Zhen Han +3
Various adaptation methods, such as LoRA, prompts, and adapters, have been proposed to enhance the performance of pre-trained vision-language models in specific domains. The robust…
GRAtt-VIS: Gated Residual Attention for Auto Rectifying Video Instance Segmentation
Tanveer Hannan, Rajat Koner, Maximilian Bernhard +5
Recent trends in Video Instance Segmentation (VIS) have seen a growing reliance on online methods to model complex and lengthy video sequences. However, the degradation of represen…
Do DALL-E and Flamingo Understand Each Other?
Hang Li, Jindong Gu, Rajat Koner +2
The field of multimodal research focusing on the comprehension and creation of both images and text has witnessed significant strides. This progress is exemplified by the emergence…
CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering
Yao Zhang, Haokun Chen, Ahmed Frikha +5
Visual Question Answering (VQA) is a multi-discipline research task. To produce the right answer, it requires an understanding of the visual content of images, the natural language…
InstanceFormer: An Online Video Instance Segmentation Framework
Rajat Koner, Tanveer Hannan, Suprosanna Shit +4
Recent transformer-based offline video instance segmentation (VIS) approaches achieve encouraging results and significantly outperform online approaches. However, their reliance on…
SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness
Jindong Gu, Hengshuang Zhao, Volker Tresp +1
Deep neural network-based image classifications are vulnerable to adversarial perturbations. The image classifications can be easily fooled by adding artificial small and impercept…
Relationformer: A Unified Framework for Image-to-Graph Generation
Suprosanna Shit, Rajat Koner, Bastian Wittmann +8
A comprehensive representation of an image requires understanding objects and their mutual relationship, especially in image-to-graph generation, e.g., road network extraction, blo…
Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answering
Rajat Koner, Hang Li, Marcel Hildebrandt +3
Visual Question Answering (VQA) is concerned with answering free-form questions about an image. Since it requires a deep semantic and linguistic understanding of the question and t…
Scenes and Surroundings: Scene Graph Generation using Relation Transformer
Rajat Koner, Poulami Sinhamahapatra, Volker Tresp
Identifying objects in an image and their mutual relationships as a scene graph leads to a deep understanding of image content. Despite the recent advancement in deep learning, the…
OODformer: Out-Of-Distribution Detection Transformer
Rajat Koner, Poulami Sinhamahapatra, Karsten Roscher +2
A serious problem in image classification is that a trained model might perform well for input data that originates from the same distribution as the data available for model train…
Capsule Network is Not More Robust than Convolutional Network
Jindong Gu, Volker Tresp, Han Hu
The Capsule Network is widely believed to be more robust than Convolutional Networks. However, there are no comprehensive comparisons between these two networks, and it is also unk…
Effective and Efficient Vote Attack on Capsule Networks
Jindong Gu, Baoyuan Wu, Volker Tresp
Standard Convolutional Neural Networks (CNNs) can be easily fooled by images with small quasi-imperceptible artificial perturbations. As alternatives to CNNs, the recently proposed…
Improving Scene Graph Classification by Exploiting Knowledge from Texts
Sahand Sharifzadeh, Sina Moayed Baharlou, Martin Schmitt +2
Training scene graph classification models requires a large amount of annotated image data. Meanwhile, scene graphs represent relational knowledge that can be modeled with symbolic…
Classification by Attention: Scene Graph Classification with Prior Knowledge
Sahand Sharifzadeh, Sina Moayed Baharlou, Volker Tresp
A major challenge in scene graph classification is that the appearance of objects and relations can be significantly different from one image to another. Previous works have addres…
Interpretable Graph Capsule Networks for Object Recognition
Jindong Gu, Volker Tresp
Capsule Networks, as alternatives to Convolutional Neural Networks, have been proposed to recognize objects from images. The current literature demonstrates many advantages of Caps…
Introspective Learning by Distilling Knowledge from Online Self-explanation
Jindong Gu, Zhiliang Wu, Volker Tresp
In recent years, many explanation methods have been proposed to explain individual classifications of deep neural networks. However, how to leverage the created explanations to imp…