Open-Sourced Reinforcement Learning Environments for Surgical Robotics

Abstract

Reinforcement Learning (RL) is a machine learning framework for artificially intelligent systems to solve a variety of complex problems. Recent years has seen a surge of successes solving challenging games and smaller domain problems, including simple though non-specific robotic manipulation and grasping tasks. Rapid successes in RL have come in part due to the strong collaborative effort by the RL community to work on common, open-sourced environment simulators such as OpenAI's Gym that allow for expedited development and valid comparisons between different, state-of-art strategies. In this paper, we aim to start the bridge between the RL and the surgical robotics communities by presenting the first open-sourced reinforcement learning environments for surgical robots, called dVRL[3]{dVRL available at https://github.com/ucsdarclab/dVRL}. Through the proposed RL environments, which are functionally equivalent to Gym, we show that it is easy to prototype and implement state-of-art RL algorithms on surgical robotics problems that aim to introduce autonomous robotic precision and accuracy to assisting, collaborative, or repetitive tasks during surgery. Learned policies are furthermore successfully transferable to a real robot. Finally, combining dVRL with the over 40+ international network of da Vinci Surgical Research Kits in active use at academic institutions, we see dVRL as enabling the broad surgical robotics community to fully leverage the newest strategies in reinforcement learning, and for reinforcement learning scientists with no knowledge of surgical robotics to test and develop new algorithms that can solve the real-world, high-impact challenges in autonomous surgery.

arXiv comment7 pages, 7 Figures

References in corpus (7)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Massively Parallel Methods for Deep Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
• PyRep: Bringing V-REP to Deep Robot Learning

Cited by in corpus (14)

• Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit
• Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer
• Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning
• Leveraging Vision and Kinematics Data to Improve Realism of Biomechanic Soft-tissue Simulation for Robotic Surgery
• Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety
• SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction
• Robotic Surgery With Lean Reinforcement Learning
• Safe Driving via Expert Guided Policy Optimization
• QEBA: Query-Efficient Boundary-Based Blackbox Attack
• Learning for Deformable Linear Object Insertion Leveraging Flexibility Estimation from Visual Cues
• Model-Predictive Control of Blood Suction for Surgical Hemostasis using Differentiable Fluid Simulations
• Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning
• Safe Exploration by Solving Early Terminated MDP
• Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot