37 citations · 59 across the 14 of their papers we have counts for
15 papers
Visual Grounding Safety in Vision-Language Models
Erfan Shayegani, Kundan Krishna, Yue Dong +3
Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet…
VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
Haz Sameen Shahgir, Xiaofu Chen, Yu Fu +4
Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perce…
From Measurement to Expertise: Empathetic Expert Adapters for Context-Based Empathy in Conversational AI Agents
Erfan Shayegani, Jina Suh, Andy Wilson +2
Empathy is a critical factor in fostering positive user experiences in conversational AI. While models can display empathy, it is often generic rather than tailored to specific tas…
Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness
Erfan Shayegani, Keegan Hines, Yue Dong +6
Computer-Use Agents (CUAs) are an increasingly deployed class of agents that take actions on GUIs to accomplish user goals. In this paper, we show that CUAs consistently exhibit Bl…
Modeling Hierarchical Thinking in Large Reasoning Models
G M Shahariar, Erfan Shayegani, Ali Nazari +1
Large Reasoning Models (LRMs) solve complex tasks by generating long Chain-of-Thought (CoT) sequences; however, the emergent dynamics governing reasoning trajectories are not well…
Evil Vizier: Vulnerabilities of LLM-Integrated XR Systems
Yicheng Zhang, Zijian Huang, Sophie Chen +3
Extended reality (XR) applications increasingly integrate Large Language Models (LLMs) to enhance user experience, scene understanding, and even generate executable XR content, and…
SENSE-7: Taxonomy and Dataset for Measuring User Perceptions of Empathy in Sustained Human-AI Conversations
Jina Suh, Lindy Le, Erfan Shayegani +5
Empathy is increasingly recognized as a key factor in human-AI communication, yet conventional approaches to "digital empathy" often focus on simulating internal, human-like emotio…
Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots
Erfan Shayegani, G M Shahariar, Sara Abdali +3
Multimodal Language Models (MMLMs) typically undergo post-training alignment to prevent harmful content generation. However, these alignment stages focus primarily on the assistant…
Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models
Saketh Bachu, Erfan Shayegani, Rohit Lal +6
Vision-language models (VLMs) have improved significantly in their capabilities, but their complex architecture makes their safety alignment challenging. In this paper, we reveal a…
Cross-Modal Safety Alignment: Is textual unlearning all you need?
Trishna Chakraborty, Erfan Shayegani, Zikui Cai +5
Recent studies reveal that integrating new modalities into Large Language Models (LLMs), such as Vision-Language Models (VLMs), creates a new attack surface that bypasses existing…
Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices
Sara Abdali, Richard Anarfi, CJ Barberan +2
Large language models (LLMs) have significantly transformed the landscape of Natural Language Processing (NLP). Their impact extends across a diverse spectrum of tasks, revolutioni…
Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu +3
Large Language Models (LLMs) are swiftly advancing in architecture and capability, and as they integrate more deeply into complex systems, the urgency to scrutinize their security…
That Doesn't Go There: Attacks on Shared State in Multi-User Augmented Reality Applications
Carter Slocum, Yicheng Zhang, Erfan Shayegani +3
Augmented Reality (AR) is expected to become a pervasive component in enabling shared virtual experiences. In order to facilitate collaboration among multiple users, it is crucial…
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models
Erfan Shayegani, Yue Dong, Nael Abu-Ghazaleh
We introduce new jailbreak attacks on vision language models (VLMs), which use aligned LLMs and are resilient to text-only jailbreak attacks. Specifically, we develop cross-modalit…
Co(ve)rtex: ML Models as storage channels and their (mis-)applications
Md Abdullah Al Mamun, Quazi Mishkatul Alam, Erfan Shayegani +3
Machine learning (ML) models are overparameterized to support generality and avoid overfitting. The state of these parameters is essentially a "don't-care" with respect to the prim…