Scalable Object Classification using Range Images

Transcription

1 Scalable Object Classification using Range Images Eunyoung Kim and Gerard Medioni Institute for Robotics and Intelligent Systems University of Southern California 1

2 What is a Range Image? Depth measurement for every pixel 2

3 Introduction Goal Development of scalable 3D free-form object recognition system using range images Motivation Various applications Scene understanding for autonomous systems HRI/HCI Limitation of image based recognition e.g. dark environment 3

4 Issues Partial surface A range image only captures the visible surface of 3D objects Occlusion and clutter Sensor limitation Noisy and inaccurate 3D points missing depth measurements 4

5 Issues Partial surface A range image only captures the visible surface of 3D objects Occlusion and clutter Sensor limitation Noisy and inaccurate 3D points missing depth measurements 4

6 Objective Fast label inference from large dataset Scalable database Incrementally learn new instances unseen before Unsupervised learning Reduce heavy reliance on human effort Robots might have to learn new objects without supervision 5

7 Overview of System Hierarchical structured database Construct a hierarchical structured database Fast object category inference/learning 6

8 Overview of System Object classification Hierarchical structured database Existing category (e.g. phone) / New class Construct a hierarchical structured database Fast object category inference/learning Object classification Infer a class of a range image representing the visible surface of an object 6

9 Overview of System Object classification Hierarchical structured database Existing category (e.g. phone) / New class New data (New instance / class) Yes Construct a hierarchical structured database Fast object category inference/learning Object classification Infer a class of a range image representing the visible surface of an object Incremental learning 6

10 Image Representation Bag of visual words Range image Visual word computation Global description: surface orientation & height Every object is in the ground coordinate system Local description: surface smoothness with neighboring regions 7

11 Image Representation Bag of visual words Range image Visual word computation Global description: surface orientation & height Every object is in the ground coordinate system Local description: surface smoothness with neighboring regions 7

12 Image Representation Bag of visual words Range image Interest point sampling(red) Visual word computation Global description: surface orientation & height Every object is in the ground coordinate system Local description: surface smoothness with neighboring regions 7

13 Image Representation Bag of visual words Range image Interest point sampling(red) Visual word computation Visual word computation Global description: surface orientation & height Every object is in the ground coordinate system Local description: surface smoothness with neighboring regions 7

14 Hierarchical Structured Database Hierarchical variant of Latent Dirichlet Allocation (LDA) model Unsupervised discovery of topic for document classification Generative model Supervised and unsupervised learning Each path in the tree = shape cluster A group of range images with similar shape pattern Co-occurrence of visual words 8

15 Hierarchical Structured Database L : Level of tree, T: # of topics 9

16 Hierarchical Structured Database L : Level of tree, T: # of topics ψ i 9

17 Hierarchical Structured Database L : Level of tree, T: # of topics For every interest point d, p(z c) ψ i 1) Sample level l i,d determine a node c the point is assigned to p(w z) 9

18 Hierarchical Structured Database L : Level of tree, T: # of topics ψ i For every interest point d, l i,d p(z c) 1) Sample level determine a node c the point is assigned to 2) Sample topic z i,d ( p(z c)) p(w z) 9

19 Hierarchical Structured Database L : Level of tree, T: # of topics ψ i For every interest point d, p(z c) 1) Sample level determine a node c the point is assigned to 2) Sample topic 3) Sample word l i,d z i,d w i,d ( p(z c)) ( p(w z i,d )) p(w z) 9

20 Hierarchical Structured Database L : Level of tree, T: # of topics ψ i For every interest point d, p(z c) 1) Sample level l i,d determine a node c the point is assigned to 2) Sample topic 3) Sample word z i,d w i,d ( p(z c)) ( p(w z i,d )) p(w z) Observation: a set of visual words Latent variables: ψ i p z i,d l i,d w i,d 9

21 Hierarchical Structured Database L : Level of tree, T: # of topics ψ i For every interest point d, p(z c) 1) Sample level l i,d determine a node c the point is assigned to 2) Sample topic 3) Sample word z i,d w i,d ( p(z c)) ( p(w z i,d )) p(w z) Observation: a set of visual words Latent variables: ψ i p z i,d l i,d w i,d Gibbs sampling: Inference of the tree structure and multinomial distributions from the given visual words 9

22 Inference of Shape Clusters Find paths with the most similar patterns with a new object j Compute p(j ψ) for every existing path p(j ψ) = p(w j,d z t )p(z t c ψ,l ) d l,t 10

23 Shape Similarity between Objects Compute similarity between objects ( 0 1) Pattern between local visual words Object = normalized histogram of pattens (576 bins) 11

24 New Object Class Given a test object j, If sim(hi, Hj) < user-defined threshold No support from object i No existing object classes support the test object New instance of a new object class Incremental learning is required!! 12

25 Incremental Learning Impractical to update the hierarchical model by batch learning Local update thanks to tree structure efficient incremental learning Infer a new instance of existing classes not properly captured in the existing paths 13

26 Approach Label: New class Existing tree tree 14 43

27 Approach Label: New class Existing tree tree Path inference process Existing tree Path candidates 14 43

28 Approach Label: New class Existing tree tree Path inference process Existing tree Path candidates 14 43

29 Approach Label: New class Existing tree Existing tree Path inference process Sample zi,d and li,d Existing tree Path candidates Existing tree Updated tree 14 43

30 Approach Label: Phone Existing tree tree 15

31 Approach Label: Phone Existing tree tree Path inference process Existing tree Path candidates 15

32 Approach Label: Phone Existing tree tree Path inference process new path? No Existing tree Path candidates 15

33 Approach Label: Phone Existing tree Existing tree Path inference process new path? Yes Sample zi,d and li,d No Existing tree Path candidates Updated Existing tree tree 15

34 Experimental Results Synthetic range images 9 classes (bottle, car, cup, desk lamp, lamp, phone, chair, plane, monitor) 1,350 (training) and 450 (test) images randomly generated from different 3D models Unlabelled 94 % recall rate Our similarity computation increases recall and precision by 16% and 14%, respectively 16

35 Real-time range sensor Swissranger SR x 144 resolution in 25 fps 88.4 % on 1,433 depth images Primesensor 640 x 480 resolution in 30 fps 89.7 % on 1,072 depth images with various view changes and occlusion 17

36 Summary Scalable framework for object classification using range images Hierarchical structured database Efficient label inference ( ms) Unsupervised incremental learning Future work Study on new object identification Spatial relationship between features / parts 18

37 Thanks! 19