40 citations · 44 across the 8 of their papers we have counts for
8 papers
VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues
Jianshu Zhang, Dongyu Yao, Renjie Pi +2
Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. De…
Personalized Visual Instruction Tuning
Renjie Pi, Jianshu Zhang, Tianyang Han +3
Recent advancements in multimodal large language models (MLLMs) have demonstrated significant progress; however, these models exhibit a notable limitation, which we refer to as "fa…
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation
KaShun Shum, Minrui Xu, Jianshu Zhang +5
Large language models (LLMs) have become increasingly prevalent in our daily lives, leading to an expectation for LLMs to be trustworthy -- - both accurate and well-calibrated (the…
Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
Renjie Pi, Jianshu Zhang, Jipeng Zhang +3
Image description datasets play a crucial role in the advancement of various applications such as image understanding, text-to-image generation, and text-image retrieval. Currently…
CORE: Mitigating Catastrophic Forgetting in Continual Learning through Cognitive Replay
Jianshu Zhang, Yankai Fu, Ziheng Peng +2
This paper introduces a novel perspective to significantly mitigate catastrophic forgetting in continuous learning (CL), which emphasizes models' capacity to preserve existing know…
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
Renjie Pi, Tianyang Han, Jianshu Zhang +6
The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs. This paper investiga…
FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
Dongyu Yao, Jianshu Zhang, Ian G. Harris +1
Jailbreak vulnerabilities in Large Language Models (LLMs), which exploit meticulously crafted prompts to elicit content that violates service guidelines, have captured the attentio…
Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition
Zi-Qiang Zhang, Jie Zhang, Jian-Shu Zhang +3
With the advance in self-supervised learning for audio and visual modalities, it has become possible to learn a robust audio-visual speech representation. This would be beneficial…