Context Aware AI Assistant and AR Interface for Lunar Extravehicular Activity (EVA) Procedural Guidance

Abstract

As human space exploration returns to the Moon, astronauts need rapid access to procedural information during extravehicular activities (EVAs), where attention is divided across navigation, repair tasks, tool handling, and environmental risk. The challenge is not the absence of information, but surfacing the right information at the right moment. We present GAIN-AI (Guided Assistant for Intelligent Navigation), a context-aware AI assistant and minimal heads-up interface for procedural guidance in simulated lunar EVA. The system operates in two layers. The first grounds a large language model with structured context: EVA procedure documents, live telemetry data, and error-handling protocols encoded as JSON. The second restructures that output into three compact units for AR display: Goal, Task, and Verification. Evaluated on 111 synthetic EVA scenarios, the system scores 10.0/10 on nominal conditions and 8.15/10 on single-fault scenarios, with performance degrading on multi-fault and boundary-threshold cases.

arXiv comment4 pages, 4 figures, 1 table. Accepted to ACM SIGGRAPH 2026 Conference Posters, July 19-23, 2026, Los Angeles, California, USA

References in corpus (6)

• Retrieval-Augmented Generation for Large Language Models: A Survey
• A Survey on Evaluation of Large Language Models
• Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
• Insights Informed Generative AI for Design: Incorporating Real-world Data for Text-to-Image Output
• 2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration
• Scene-Aware Urban Design: A Human-AI Recommendation Framework Using Co-Occurrence Embeddings and Vision-Language Models