21 citations · 173 across the 53 of their papers we have counts for
36 papers · 1 filter
AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models
Haokun Chen, Jianing Li, Yao Zhang +4
Multimodal Large Language Models (MLLMs) achieve impressive performance once optimized on massive datasets. Such datasets often contain sensitive or copyrighted content, raising si…
True Multimodal In-Context Learning Needs Attention to the Visual Context
Shuo Chen, Jianzhe Liu, Zhen Han +5
Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks from a few multimodal demon…
Multimodal Pragmatic Jailbreak on Text-to-image Models
Tong Liu, Zhixin Lai, Jiawen Wang +6
Diffusion models have recently achieved remarkable advancements in terms of image quality and fidelity to textual prompts. Concurrently, the safety of such generative models has be…
Localizing Events in Videos with Multimodal Queries
Gengyuan Zhang, Mang Ling Ada Fok, Jialu Ma +5
Localizing events in videos based on semantic queries is a pivotal task in video understanding, with the growing significance of user-oriented applications like video search. Yet,…
Which Model Generated This Image? A Model-Agnostic Approach for Origin Attribution
Fengyuan Liu, Haochen Luo, Yiming Li +2
Recent progress in visual generative models enables the generation of high-quality images. To prevent the misuse of generated images, it is important to identify the origin model t…
Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image
Zefeng Wang, Zhen Han, Shuo Chen +6
Multimodal LLMs (MLLMs) with a great ability of text and image understanding have received great attention. To achieve better reasoning with MLLMs, Chain-of-Thought (CoT) reasoning…
SPOT! Revisiting Video-Language Models for Event Understanding
Gengyuan Zhang, Jinhe Bi, Jindong Gu +2
Understanding videos is an important research topic for multimodal learning. Leveraging large-scale datasets of web-crawled video-text pairs as weak supervision has become a pre-tr…
MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
Xin Liu, Yichen Zhu, Jindong Gu +3
The security concerns surrounding Large Language Models (LLMs) have been extensively explored, yet the safety of Multimodal Large Language Models (MLLMs) remains understudied. In t…
Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation
Hang Li, Chengzhi Shen, Philip Torr +2
Diffusion-based models have gained significant popularity for text-to-image generation due to their exceptional image-generation capabilities. A risk with these models is the poten…
Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks
Jindong Gu, Fangyun Wei, Philip Torr +1
Deep Neural Networks can be easily fooled by small and imperceptible perturbations. The query-based black-box attack (QBBA) is able to create the perturbations using model output p…
Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averaging
Xiaojun Jia, Yuefeng Chen, Xiaofeng Mao +5
Fast Adversarial Training (FAT) not only improves the model robustness but also reduces the training cost of standard adversarial training. However, fast adversarial training often…
Multi-event Video-Text Retrieval
Gengyuan Zhang, Jisen Ren, Jindong Gu +1
Video-Text Retrieval (VTR) is a crucial multi-modal task in an era of massive video-text data on the Internet. A plethora of work characterized by using a two-stream Vision-Languag…
Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression
Xuanlong Yu, Gianni Franchi, Jindong Gu +1
Uncertainty quantification is critical for deploying deep neural networks (DNNs) in real-world applications. An Auxiliary Uncertainty Estimator (AuxUE) is one of the most effective…
Reliable Evaluation of Adversarial Transferability
Wenqian Yu, Jindong Gu, Zhijiang Li +1
Adversarial examples (AEs) with small adversarial perturbations can mislead deep neural networks (DNNs) into wrong predictions. The AEs created on one DNN can also fool another DNN…
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
Shuo Chen, Jindong Gu, Zhen Han +3
Various adaptation methods, such as LoRA, prompts, and adapters, have been proposed to enhance the performance of pre-trained vision-language models in specific domains. The robust…
Towards Robust Prompts on Vision-Language Models
Jindong Gu, Ahmad Beirami, Xuezhi Wang +3
With the advent of vision-language models (VLMs) that can perform in-context and prompt-based learning, how can we design prompting approaches that robustly generalize to distribut…
Backdoor Defense via Adaptively Splitting Poisoned Dataset
Kuofeng Gao, Yang Bai, Jindong Gu +2
Backdoor defenses have been studied to alleviate the threat of deep neural networks (DNNs) being backdoor attacked and thus maliciously altered. Since DNNs usually adopt some exter…
Influencer Backdoor Attack on Semantic Segmentation
Haoheng Lan, Jindong Gu, Philip Torr +1
When a small number of poisoned samples are injected into the training dataset of a deep neural network, the network can be induced to exhibit malicious behavior during inferences,…
Explainability and Robustness of Deep Visual Classification Models
Jindong Gu
In the computer vision community, Convolutional Neural Networks (CNNs), first proposed in the 1980's, have become the standard visual classification model. Recently, as alternative…
Do DALL-E and Flamingo Understand Each Other?
Hang Li, Jindong Gu, Rajat Koner +2
The field of multimodal research focusing on the comprehension and creation of both images and text has witnessed significant strides. This progress is exemplified by the emergence…
CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering
Yao Zhang, Haokun Chen, Ahmed Frikha +5
Visual Question Answering (VQA) is a multi-discipline research task. To produce the right answer, it requires an understanding of the visual content of images, the natural language…
Towards Efficient Adversarial Training on Vision Transformers
Boxi Wu, Jindong Gu, Zhifeng Li +3
Vision Transformer (ViT), as a powerful alternative to Convolutional Neural Network (CNN), has received much attention. Recent work showed that ViTs are also vulnerable to adversar…
Watermark Vaccine: Adversarial Attacks to Prevent Watermark Removal
Xinwei Liu, Jian Liu, Yang Bai +4
As a common security tool, visible watermarking has been widely applied to protect copyrights of digital images. However, recent works have shown that visible watermarks can be rem…
SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness
Jindong Gu, Hengshuang Zhao, Volker Tresp +1
Deep neural network-based image classifications are vulnerable to adversarial perturbations. The image classifications can be easily fooled by adding artificial small and impercept…