RL-LABEL: A Deep Reinforcement Learning Approach Intended for AR Label Placement in Dynamic Scenarios

Abstract

Labels are widely used in augmented reality (AR) to display digital information. Ensuring the readability of AR labels requires placing them occlusion-free while keeping visual linkings legible, especially when multiple labels exist in the scene. Although existing optimization-based methods, such as force-based methods, are effective in managing AR labels in static scenarios, they often struggle in dynamic scenarios with constantly moving objects. This is due to their focus on generating layouts optimal for the current moment, neglecting future moments and leading to sub-optimal or unstable layouts over time. In this work, we present RL-LABEL, a deep reinforcement learning-based method for managing the placement of AR labels in scenarios involving moving objects. RL-LABEL considers the current and predicted future states of objects and labels, such as positions and velocities, as well as the user's viewpoint, to make informed decisions about label placement. It balances the trade-offs between immediate and long-term objectives. Our experiments on two real-world datasets show that RL-LABEL effectively learns the decision-making process for long-term optimization, outperforming two baselines (i.e., no view management and a force-based method) by minimizing label occlusions, line intersections, and label movement distance. Additionally, a user study involving 18 participants indicates that RL-LABEL excels over the baselines in aiding users to identify, compare, and summarize data on AR labels within dynamic scenes.

References in corpus (13)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Unity: A General Platform for Intelligent Agents
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
• A Survey of Deep Reinforcement Learning in Video Games
• Towards an Understanding of Situated AR Visualization for Basketball Free-Throw Training
• MARVisT: Authoring Glyph-based Visualization in Mobile Augmented Reality
• Augmenting Static Visualizations with PapARVis Designer
• Labeling Out-of-View Objects in Immersive Analytics to Support Situated Visual Searching
• iBall: Augmenting Basketball Videos with Gaze-moderated Embedded Visualizations
• Table2Charts: Recommending Charts by Learning Shared Table Representations
• Context-Responsive Labeling in Augmented Reality
• Evolutionary Deep Reinforcement Learning Using Elite Buffer: A Novel Approach Towards DRL Combined with EA in Continuous Control Tasks

Cited by in corpus (2)

• Adaptive 3D UI Placement in Mixed Reality Using Deep Reinforcement Learning
• InSituTale: Enhancing Augmented Data Storytelling with Physical Objects