2 citations · 3 across the 11 of their papers we have counts for
12 papers
Make Your LVLM KV Cache More Lightweight
Xihao Chen, Yangyang Guo, Roger Zimmermann
Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (L…
LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
Yangyang Guo, Ziwei Xu, Si Liu +2
This study reveals a previously unexplored vulnerability in the safety alignment of Large Language Models (LLMs). Existing aligned LLMs predominantly respond to unsafe queries with…
Involuntary Jailbreak: On Self-Prompting Attacks
Yangyang Guo, Yangyan Li, Mohan Kankanhalli
In this study, we disclose a worrying new vulnerability in Large Language Models (LLMs), which we term \textbf{involuntary jailbreak}. Unlike existing jailbreak attacks, this weakn…
FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
Sizhe Dang, Yangyang Guo, Yanjun Zhao +4
Fine-tuning large language models (LLMs) often faces GPU memory bottlenecks: the backward pass of first-order optimizers like Adam increases memory usage to more than 10 times the…
Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds
Yang Guo, Yutian Tao, Yifei Ming +2
Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostl…
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Yong Zhang, Heng Li, Yanwen Huang +6
Retrieval-augmented generation (RAG) often suffers from long and noisy retrieved contexts. Existing context compression methods typically rely on heuristic relevance estimation or…
VidLBEval: Benchmarking and Mitigating Language Bias in Video-Involved LVLMs
Yiming Yang, Yangyang Guo, Hui Lu +1
Recently, Large Vision-Language Models (LVLMs) have made significant strides across diverse multimodal tasks and benchmarks. This paper reveals a largely under-explored problem fro…
Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM
Yangyang Guo, Ziwei Xu, Xilie Xu +3
This technical report introduces our top-ranked solution that employs two approaches, \ie suffix injection and projected gradient descent (PGD) , to address the TiFA workshop MLLM…
Joint Vision-Language Social Bias Removal for CLIP
Haoyu Zhang, Yangyang Guo, Mohan Kankanhalli
Vision-Language (V-L) pre-trained models such as CLIP show prominent capabilities in various downstream tasks. Despite this promise, V-L models are notoriously limited by their inh…
SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency
Yangyang Guo, Mohan Kankanhalli
While contrastive pre-training is widely employed, its data efficiency problem has remained relatively under-explored thus far. Existing methods often rely on static coreset select…
VidHal: Benchmarking Temporal Hallucinations in Vision LLMs
Wey Yeh Choong, Yangyang Guo, Mohan Kankanhalli
Vision Large Language Models (VLLMs) are widely acknowledged to be prone to hallucinations. Existing research addressing this problem has primarily been confined to image inputs, w…
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning
Jifan Zhang, Lalit Jain, Yang Guo +9
We present a novel multimodal preference dataset for creative tasks, consisting of over 250 million human ratings on more than 2.2 million captions, collected through crowdsourcing…