Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusion

Abstract

Endoscopic Submucosal Dissection (ESD) is a well-established technique for removing epithelial lesions. Predicting dissection trajectories in ESD videos offers significant potential for enhancing surgical skill training and simplifying the learning process, yet this area remains underexplored. While imitation learning has shown promise in acquiring skills from expert demonstrations, challenges persist in handling uncertain future movements, learning geometric symmetries, and generalizing to diverse surgical scenarios. To address these, we introduce a novel approach: Implicit Diffusion Policy with Equivariant Representations for Imitation Learning (iDPOE). Our method models expert behavior through a joint state action distribution, capturing the stochastic nature of dissection trajectories and enabling robust visual representation learning across various endoscopic views. By incorporating a diffusion model into policy learning, iDPOE ensures efficient training and sampling, leading to more accurate predictions and better generalization. Additionally, we enhance the model's ability to generalize to geometric symmetries by embedding equivariance into the learning process. To address state mismatches, we develop a forward-process guided action inference strategy for conditional sampling. Using an ESD video dataset of nearly 2000 clips, experimental results show that our approach surpasses state-of-the-art methods, both explicit and implicit, in trajectory prediction. To the best of our knowledge, this is the first application of imitation learning to surgical skill development for dissection trajectory prediction.

References in corpus (16)

• Denoising Diffusion Probabilistic Models
• Surgical Data Science: Enabling Next-Generation Surgery
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• 2018 Robotic Scene Segmentation Challenge
• Incorporating Symmetry into Deep Dynamics Models for Improved Generalization
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Is Conditional Generative Modeling all you need for Decision-Making?
• Behavior Transformers: Cloning $k$ modes with one stone
• Automatic Symmetry Discovery with Lie Algebra Convolutional Network
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• The MAGICAL Benchmark for Robust Imitation
• Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization
• Learning Layer-wise Equivariances Automatically using Gradients