SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation

Abstract

High-resolution representation is essential for achieving good performance in human pose estimation models. To obtain such features, existing works utilize high-resolution input images or fine-grained image tokens. However, this dense high-resolution representation brings a significant computational burden. In this paper, we address the following question: "Only sparse human keypoint locations are detected for human pose estimation, is it really necessary to describe the whole image in a dense, high-resolution manner?" Based on dynamic transformer models, we propose a framework that only uses Sparse High-resolution Representations for human Pose estimation (SHaRPose). In detail, SHaRPose consists of two stages. At the coarse stage, the relations between image regions and keypoints are dynamically mined while a coarse estimation is generated. Then, a quality predictor is applied to decide whether the coarse estimation results should be refined. At the fine stage, SHaRPose builds sparse high-resolution representations only on the regions related to the keypoints and provides refined high-precision human pose estimations. Extensive experiments demonstrate the outstanding performance of the proposed method. Specifically, compared to the state-of-the-art method ViTPose, our model SHaRPose-Base achieves 77.4 AP (+0.5 AP) on the COCO validation set and 76.7 AP (+0.5 AP) on the COCO test-dev set, and infers at a speed of $1.4\times$ faster than ViTPose-Base.

Accepted to AAAI 2024

References in corpus (11)

• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
• QuadTree Attention for Vision Transformers
• Token Merging: Your ViT But Faster
• TFPose: Direct Human Pose Estimation with Transformers
• Distribution-Aware Coordinate Representation for Human Pose Estimation
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation
• CF-ViT: A General Coarse-to-Fine Method for Vision Transformer
• Towards High Performance One-Stage Human Pose Estimation