SuperAnimal pretrained pose estimation models for behavioral analysis

Abstract

Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models that can be used on over 45 species, without additional human labels. Concretely, we introduce a method to unify the keypoint space across differently labeled datasets (via our generalized data converter) and for training these diverse datasets in a manner such that they don't catastrophically forget keypoints given the unbalanced inputs (via our keypoint gradient masking and memory replay approaches). These models show excellent performance across six pose benchmarks. Then, to ensure maximal usability for end-users, we demonstrate how to fine-tune the models on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If the models are fine-tuned, we show SuperAnimal models are 10-100$\times$ more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification in mice and gait analysis in horses. Collectively, this presents a data-efficient solution for animal pose estimation.

arXiv commentModels and demos available at http://modelzoo.deeplabcut.org

References in corpus (17)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Overcoming catastrophic forgetting in neural networks
• Learning Transferable Visual Models From Natural Language Supervision
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Seeing biodiversity: perspectives in machine learning for wildlife conservation
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• Deep learning tools for the measurement of animal behavior in neuroscience
• Learnable latent embeddings for joint behavioral and neural analysis
• Deep High-Resolution Representation Learning for Visual Recognition
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• SuperAnimal pretrained pose estimation models for behavioral analysis
• Measuring and modeling the motor system with machine learning
• AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild
• APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking
• Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation
• MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior
• If your data distribution shifts, use self-learning

Cited by in corpus (2)

• SuperAnimal pretrained pose estimation models for behavioral analysis
• Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems