885 citations · 1.7k across the 71 of their papers we have counts for
91 papers
Kernel-Based Steering of CLIP with Vision-Language Model Preferences
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Sina Mansouri +3
Large vision-language models (VLMs) can judge visual similarity, but their judgments are not directly available as compact image embeddings for efficient comparison. We study how t…
Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases
Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh +1
Prompt learning adapts CLIP to downstream recognition by replacing hand-written templates with learned continuous context vectors, which in Context Optimization (CoOp) form a dense…
ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1
Prompt tuning adapts vision--language models with few trainable parameters, but computing their gradients still requires backpropagation through frozen encoders. We introduce ZOMP,…
Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2
Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): it has no single correct answer, and most aesthetic evaluation measures models against nu…
REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations
Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh +1
Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standar…
Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment
Zhiyu Xue, Zimo Qi, Guangliang Liu +2
Safety alignment aims to ensure that large language models (LLMs) refuse harmful requests by post-training on harmful queries paired with refusal answers. Although safety alignment…
MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1
Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending pro…
INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models
Parsa Madinei, Ryan Solgi, Ziqi Wen +3
We introduce INTERLACE, a novel framework that prunes redundant layers in VLMs while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lea…
pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
Sajjad Ghiasvand, Mahnoosh Alizadeh, Ramtin Pedarsani
Vision-Language Models (VLMs) like CLIP have demonstrated remarkable generalization in zero- and few-shot settings, but adapting them efficiently to decentralized, heterogeneous da…
Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations
Zhiyu Xue, Reza Abbasi-Asl, Ramtin Pedarsani
Generative medical vision-language models~(Med-VLMs) are primarily designed to generate complex textual information~(e.g., diagnostic reports) from multimodal inputs including visi…
Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1
Vision-Language Models (VLMs) such as CLIP have shown remarkable performance in cross-modal tasks through large-scale contrastive pre-training. To adapt these large transformer-bas…
The Safety-Privacy Tradeoff in Linear Bandits
Arghavan Zibaie, Spencer Hutchinson, Ramtin Pedarsani +1
We consider a collection of linear stochastic bandit problems, each modeling the random response of different agents to proposed interventions, coupled together by a global safety…
SPEX: Scaling Feature Interaction Explanations for LLMs
Justin Singh Kang, Landon Butler, Abhineet Agarwal +4
Large language models (LLMs) have revolutionized machine learning due to their ability to capture complex interactions between input features. Popular post-hoc explanation methods…
Decentralized Low-Rank Fine-Tuning of Large Language Models
Sajjad Ghiasvand, Mahnoosh Alizadeh, Ramtin Pedarsani
While parameter-efficient fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) offer computationally efficient adaptations of Large Language Models (LLMs), their practical…
No Free Lunch for Defending Against Prefilling Attack by In-Context Learning
Zhiyu Xue, Guangliang Liu, Bocheng Chen +2
The security of Large Language Models (LLMs) has become an important research topic since the emergence of ChatGPT. Though there have been various effective methods to defend again…
Multi-Bin Batching for Increasing LLM Inference Throughput
Ozgur Guldogan, Jackson Kunde, Kangwook Lee +1
As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LL…
Conflict-Aware Adversarial Training
Zhiyu Xue, Haohan Wang, Yao Qin +1
Adversarial training is the most effective method to obtain adversarial robustness for deep neural networks by directly involving adversarial samples in the training procedure. To…
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models
Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue +3
Parameter-efficient fine-tuning (PEFT) methods typically assume that Large Language Models (LLMs) are trained on data from a single device or client. However, real-world scenarios…
Long-Term Fairness in Sequential Multi-Agent Selection with Positive Reinforcement
Bhagyashree Puranik, Ozgur Guldogan, Upamanyu Madhow +1
While much of the rapidly growing literature on fair decision-making focuses on metrics for one-shot decisions, recent work has raised the intriguing possibility of designing seque…
Robust Decentralized Learning with Local Updates and Gradient Tracking
Sajjad Ghiasvand, Amirhossein Reisizadeh, Mahnoosh Alizadeh +1
As distributed learning applications such as Federated Learning, the Internet of Things (IoT), and Edge Computing grow, it is critical to address the shortcomings of such technolog…
Generalization Properties of Adversarial Training for -Bounded Adversarial Attacks
Payam Delgosha, Hamed Hassani, Ramtin Pedarsani
We have widely observed that neural networks are vulnerable to small additive perturbations to the input causing misclassification. In this paper, we focus on the -bounded…
Learning to Understand: Identifying Interactions via the Möbius Transform
Justin S. Kang, Yigit E. Erginbas, Landon Butler +2
One of the key challenges in machine learning is to find interpretable representations of learned functions. The Möbius transform is essential for this purpose, as its coefficients…
Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
Mark Beliaev, Ramtin Pedarsani
In Imitation Learning (IL), utilizing suboptimal and heterogeneous demonstrations presents a substantial challenge due to the varied nature of real-world data. However, standard IL…
Pricing for Multi-modal Pickup and Delivery Problems with Heterogeneous Users
Mark Beliaev, Negar Mehr, Ramtin Pedarsani
In this paper, we study the pickup and delivery problem with multiple transportation modalities, and address the challenge of efficiently allocating transportation resources while…