A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

Abstract

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.

arXiv commentReview, 21 pages, 8 figures and 5 boxes

References in corpus (34)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• scikit-image: Image processing in Python
• Representation Learning with Contrastive Predictive Coding
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Shortcut Learning in Deep Neural Networks
• Learning deep representations by mutual information estimation and maximization
• A guide to convolution arithmetic for deep learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Visualizing the Loss Landscape of Neural Nets
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Contrastive Multiview Coding
• Deep learning tools for the measurement of animal behavior in neuroscience
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• An efficient framework for learning sentence representations
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Self-supervised Learning of Motion Capture
• Rethinking ImageNet Pre-training
• Movement science needs different pose tracking algorithms
• Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos
• An Analysis of Pre-Training on Object Detection
• MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
• Mechanical principles of dynamic terrestrial self-righting using wings
• ArtTrack: Articulated Multi-person Tracking in the Wild

Cited by in corpus (11)

• Seeing biodiversity: perspectives in machine learning for wildlife conservation
• SuperAnimal pretrained pose estimation models for behavioral analysis
• Perspectives on individual animal identification from biology and computer vision
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Measuring and modeling the motor system with machine learning
• AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild
• Machine Learning for Optical Motion Capture-driven Musculoskeletal Modelling from Inertial Motion Capture Data
• Deep-Learning-Based Markerless Pose Estimation Systems in Gait Analysis: DeepLabCut Custom Training and the Refinement Function
• End-to-End Trainable Multi-Instance Pose Estimation with Transformers
• BlanketGen - A synthetic blanket occlusion augmentation pipeline for MoCap datasets
• Automated Tracking of Primate Behavior