Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy

Abstract

Advances in artificial intelligence and human-computer interaction will likely lead to extended reality (XR) becoming pervasive. While XR can provide users with interactive, engaging, and immersive experiences, non-player characters are often utilized in pre-scripted and conventional ways. This paper argues for using large language models (LLMs) in XR by embedding them in avatars or as narratives to facilitate inclusion through prompt engineering and fine-tuning the LLMs. We argue that this inclusion will promote diversity for XR use. Furthermore, the versatile conversational capabilities of LLMs will likely increase engagement in XR, helping XR become ubiquitous. Lastly, we speculate that combining the information provided to LLM-powered spaces by users and the biometric data obtained might lead to novel privacy invasions. While exploring potential privacy breaches, examining user privacy concerns and preferences is also essential. Therefore, despite challenges, LLM-powered XR is a promising area with several opportunities.

ACM Conversational User Interfaces 2024

References in corpus (23)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• PaLM: Scaling Language Modeling with Pathways
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• PaLM 2 Technical Report
• Scaling Speech Technology to 1,000+ Languages
• "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents
• A privacy-preserving approach to streaming eye-tracking data
• Digital Transformations of Classrooms in Virtual Reality
• Differential Privacy for Eye Tracking with Temporal Correlations
• Cognitive Mirage: A Review of Hallucinations in Large Language Models
• Going Incognito in the Metaverse: Achieving Theoretically Optimal Privacy-Usability Tradeoffs in VR
• ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges
• Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey
• LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
• Privacy Issues in Large Language Models: A Survey
• Deep Motion Masking for Secure, Usable, and Scalable Real-Time Anonymization of Virtual Reality Motion Data
• Towards Everyday Virtual Reality through Eye Tracking
• InferFix: End-to-End Program Repair with LLMs

Cited by in corpus (5)

• How LLMs are Shaping the Future of Virtual Reality
• Exploring Collaborative GenAI Agents in Synchronous Group Settings: Eliciting Team Perceptions and Design Considerations for the Future of Work
• Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes
• Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curling
• CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR