14 papers
Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality
Elias Rotondo, Lin Duan, Yanming Xiu +3
Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare delivery, and risk-mitigatio…
Through Human Eyes and Machine Eyes: Understanding View Mismatch in Video See-Through Extended Reality
Yanming Xiu
Video see-through extended reality (VST XR) systems commonly use headset screenshots or captured frames as proxies for the user's first-person visual context. However, the system-c…
Real-Time Visual Obstruction Detection in Surgical Augmented Reality
Shih-Chin Yang, Yanming Xiu, Hanting Ye +3
Surgical augmented reality (AR) can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace. However, the v…
Is It Real? Exploiting Virtual-Physical Discrimination Vulnerability in Mixed Reality
Xueyang Wang, Xihuan Yao, Yanming Xiu +3
Consumer mixed reality (MR) headsets seamlessly blend virtual content into physical environments with sufficient fidelity that users may be unable to distinguish virtual objects fr…
Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality
Yanming Xiu, Zhengyuan Jiang, Neil Zhenqiang Gong +1
Augmented reality (AR) has rapidly expanded over the past decade. As AR becomes increasingly integrated into daily life, its security and reliability emerge as critical challenges.…
User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environments
Junfeng Lin, Yanming Xiu, Maria Gorlatova
Open-set object detection (OSOD) localizes objects while identifying and rejecting unknown classes at inference. While recent OSOD models perform well on benchmarks, their behavior…
Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?
Sazia Tabasum Mim, Jack Morris, Manish Dhakal +3
To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason…
Demonstrating Visual Information Manipulation Attacks in Augmented Reality: A Hands-On Miniature City-Based Setup
Yanming Xiu, Maria Gorlatova
Augmented reality (AR) enhances user interaction with the real world but also presents vulnerabilities, particularly through Visual Information Manipulation (VIM) attacks. These at…
Say It, See It: A Systematic Evaluation on Speech-Based 3D Content Generation Methods in Augmented Reality
Yanming Xiu, Joshua Chilukuri, Shunav Sen +1
As augmented reality (AR) applications increasingly require 3D content, generative pipelines driven by natural input such as speech offer an alternative to manual asset creation. I…
A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Reality
Rongqian Chen, Allison Andreyev, Yanming Xiu +7
Augmented Reality (AR) enriches human perception by overlaying virtual elements onto the physical world. However, this tight coupling between virtual and real content makes AR vuln…
Toward Safe, Trustworthy and Realistic Augmented Reality User Experience
Yanming Xiu
As augmented reality (AR) becomes increasingly integrated into everyday life, ensuring the safety and trustworthiness of its virtual content is critical. Our research addresses the…
Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approach
Yanming Xiu, Maria Gorlatova
The virtual content in augmented reality (AR) can introduce misleading or harmful information, leading to semantic misunderstandings or user errors. In this work, we focus on visua…
Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble
Lin Duan, Yanming Xiu, Maria Gorlatova
Augmented Reality (AR) enhances the real world by integrating virtual content, yet ensuring the quality, usability, and safety of AR experiences presents significant challenges. Co…
ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality
Yanming Xiu, Tim Scargill, Maria Gorlatova
In Augmented Reality (AR), virtual content enhances user experience by providing additional information. However, improperly positioned or designed virtual content can be detriment…