SAPIEN: Affective Virtual Agents Powered by Large Language Models

Abstract

In this demo paper, we introduce SAPIEN, a platform for high-fidelity virtual agents driven by large language models that can hold open domain conversations with users in 13 different languages, and display emotions through facial expressions and voice. The platform allows users to customize their virtual agent's personality, background, and conversation premise, thus providing a rich, immersive interaction experience. Furthermore, after the virtual meeting, the user can choose to get the conversation analyzed and receive actionable feedback on their communication skills. This paper illustrates an overview of the platform and discusses the various application domains of this technology, ranging from entertainment to mental health, communication training, language learning, education, healthcare, and beyond. Additionally, we consider the ethical implications of such realistic virtual agent representations and the potential challenges in ensuring responsible use.

References in corpus (8)

• Training language models to follow instructions with human feedback
• GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
• The Microsoft 2017 Conversational Speech Recognition System
• Tacotron: Towards End-to-End Speech Synthesis
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• A Virtual Conversational Agent for Teens with Autism: Experimental Results and Design Lessons
• Recent Advances in End-to-End Automatic Speech Recognition
• FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition

Cited by in corpus (3)

• The influence of persona and conversational task on social interactions with a LLM-controlled embodied conversational agent
• Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agents
• Estuary: A Framework For Building Multimodal Low-Latency Real-Time Socially Interactive Agents