Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

Transcription

1 Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields Authors: Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh Presented by: Suraj Kesavan, Priscilla Jennifer ECS 289G: Visual Recognition 02/27/2018

2 Introduction to Pose Estimation and Association

3 Challenges Unknown number of people that can occur in a frame. Complex Spatial Interference - Contact, Occlusion between people. Variance in person scales Run time Complexity.

4 Top-down Approach: Person Detection + Pose Estimation Faster R-CNN (Person Detector) Papandreou, George, et al. "Towards accurate multiperson pose estimation in the wild." arxiv preprint arxiv: (2017). ResNet

5 Bottom-up Approach: Parts Detection and Parts Association Parts Detection Image CNN Parts Association

6 Sub-network 1: Part Detection S = (S1, S2, SJ), Si is a confidence map - for each part (j )

7 Sub-network 1: Part Detection p - (x, y) in an image W(p) - Binary mask {0, 1} Stj(p) - Confidence score for joint J {1.J} at stage t S*j(p) - Ground truth confidence map xj,k - Ground truth of body part j for person k S*j,k(p) - confidence score for joint j for person k Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No

8 Sub-network 1: Parts Detection

9 Sub-network 2: Part Association using Part Affinity Fields Part Affinity Fields encodes Orientation

10 Sub-network 2: Part Association using Part Affinity Fields L = (L1, L2, LC), Li is a vector field - for each limb (c ) v - normalized unit vector along Xj1,kXj2,k lc.k - distance between Xj1,kXj2,k σl - limb width nc(p) - No of non-zero vectors at point p for all k people Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No

11 CNN Architecture

12 Sequential Prediction with Learned Spatial Context Stage 1 P CNN Stage T Stage 2 P P CNN CNN pose%20estimation-cmu.pdf Right Wrist - Stage 1 Right Wrist - Stage 2 Right Wrist - Stage T

13 Jointly Learning Parts Detection and Parts Association Stage 1 Stage 2 P CNN 2nd Branch Part Affinity Fields Stage T P P CNN CNN Stage 2 Stage T P CNN CNN

14 OpenPose Pipeline

15 Testing - Non-maximum Suppression NMS 104fc8

16 Testing - Line Integral PAF l1 n1 n2 n3 n4 0/1 0/1 0/1 0/1 l2 0/1 0/1 0/1 0/1 l3 0/1 0/1 0/1 0/1 Bipartite graph Line integral n1 n2 n3 n4 l l l Weighted Bipartite graph

17 Midpoint Score Map for Part-to-Part Association

18 Spatial Ambiguity of the Midpoint Representation Correct Connection Wrong Connection

19 Increasing Midpoint Number Cannot Solve The Problem Correct Connection Wrong Connection

20 Part Affinity Fields Avoid Spatial Ambiguity Elbow Wrist Correct Connection Wrong Connection

21 Greedy algorithm for Graph matching Shoulder Elbow Wrist

22 Hungarian algorithm for Graph matching Shoulder E1 E2 S S E1 E2 W W Elbow Wrist

23 Hungarian algorithm for Graph matching Shoulder E1 E2 S S Elbow Maximum E = = E1 E2 W W Wrist

24 Hungarian algorithm for Graph matching Shoulder E1 E2 S S Elbow Maximum E = = E1 E2 W W Wrist

25

26 Results on the MPII Multi Person Dataset Comparison of map across other implementations on MPII Dataset. Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No Comparison of the graph matching algorithms on validation set.

27 Results on the MPII Multi Person Dataset map curves of different experiments Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No map curves at different stages of the experiment.

28 Results on COCO Challenge Validation Set Comparison of results from the top-down approach with this approach. Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No Comparison of techniques which use Convolutional Pose machines(cpm) with this approach.

29 Strength Robustness to early commitment Run time for this method only increases slowly with the no of people in the image. Runtime Analysis Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No

30 Weakness Failure cases Greedy algorithm can fail to give a perfect matching. It fails in certain cases of rare posture, false positives for statues, overlapping limbs. Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No

31

32 Questions and Discussion.