Multimodal detection and recognition of persons with a static robot

Transcription

1 Multimodal detection and recognition of persons with a static robot Jaldert Rombouts rombouts@ai.rug.nl Internal advisors: prof. dr. L.R.B Schomaker. Artificial Intelligence, University of Groningen drs. T. van der Zant. Artificial Intelligence, University of Groningen External advisor: dr. P. E. Rybski. Robotics Institute, Carnegie Mellon University

2 Overview Introduction Background and Approach Experiments and Results Discussion Questions

3 Introduction SnackBot (Lee et al., 2009) Human-Robot Interaction (HRI) Vending machine Topic: Person detection and recognition

4 Introduction Solutions: ID-Cards, biometrics (Jain et al., 2004b) Disadvantage: Close proximity, conscious user effort More natural solution? Based on soft biometrics (Jain et al., 2004a) Color, gait, shape (combinations) Passive (e.g. camera)

5 Introduction Implemented soft-biometric system(s) based on related work Evaluated performance: Multiple poses Various distances

6 Background and Approach Segmentation Feature extraction Data set First: robot and sensors

7 Robot and Sensors 8 7 Model Foreground COG 6 5 X (meters) Y (meters)

8 Segmentation Implemented two methods: 1. Background modeling based (Horprasert et al. (1999)) 2. Stereo based (Darrell et al. (2000); Zhao et al. (2000)) Combined with laser-based leg-detector

9 Feature extraction Color: (HS)V, nrgb, Y(CrCb), CIE-L(ab) Torso or Head + Torso (spatial histogram) 1. Mean + standard deviation 2. 1D and 2D chromaticity histograms 4, 8, 16, 32 bins Person Height: from stereo

10 Data set 30 persons (fairly large w.r.t. related work) 2 environments, 9 positions, 4 poses Repeated recording for validation

11 Low office dividers Chair Opening Drawers Experimenter Desk 510 meter High office dividers Table Chair Boxes Legend Point in cluttered scene Point in sparse scene 710 meter

12 Low office dividers Chair Opening Drawers Experimenter Desk 510 meter High office dividers Table Chair Boxes Legend Point in cluttered scene Point in sparse scene 710 meter

13 Low office dividers Chair Opening Drawers Experimenter Desk 510 meter High office dividers Table Chair Boxes Legend Point in cluttered scene Point in sparse scene 710 meter

14 Data set - Poses

15 Data set - Poses

16

17 Overview Introduction Background and Approach Experiments and Results Discussion Questions

18 Recognition Main questions: What is the best set of features for recognition? Robustness against pose and distance? Difference between segmentation methods? Difference between environments?

19 Testing method Classifiers: K-Nearest Neighbor (knn) Support Vector Machine (SVM) Random Forest (RF) (Breiman, 2001) Good performance on large featurevectors with low information features

20 Testing method Cross-validation Average CA over environments

21 Recognition Feature selection: 1. Color space 2. Color feature 3. Combining height Detailed experiments: Environment, location and pose

22 Color features (1+2) HSV 2D 32 bin histogram was best Features extracted from torso slightly better than head+torso ±0.55 for DS and ±0.64 for BGM (Baseline ~0.03)

23 Bin size vs. CA knn RF SVM CA Bin size

24 Combining height Not trivial: knn and SVM use distances in feature-space Small impact single feature Idea: make height more important by scaling axis

25 SVM-DS SVM-BGM knn-ds knn-bgm RF-DS RF-BGM 0.7 CA Scaling Factor

26 Combining height SVM and knn profit from height (.15 in CA) RF only marginally (±0.01): overfitting Height does not seem important at trainlocation, but gains importance with distance RF cannot make use of domain-knowledge designer

27 Detailed experiments 1. Environment 2. Position 3. Pose

28 Position Clear influence of distance: (DS) and (BGM) [close] (DS) and (BGM) [medium] (DS) and (BGM) [far] Scores BGM/DS very similar for locations 1-6, BGM better at 7-9

29 Pose Robustness to varying pose Train on single pose (e.g. front), test on all poses: Per location (all four poses) 1 vs. 4: ±0.10 drop in CA when averaged over all locations

30 Summary Simple features yield good performance BGM better than DS (esp. further locations) Little influence of environment Clear influence of position (distance) Reasonably robust to pose

31 Discussion Careful with generalization: e.g. HSV might be best in our experimental circumstances, but worse in others Fitted to subjects/environments? Try more environments/subjects

32 Discussion Intra-day recognition only (Darrell et al. 2000; Harville, 2005) Combine with e.g. face, voice No unsupervised enrollment of users

33 Questions?

34 References Breiman, L. (2001). Random forests. Machine learning, 45(1):5 32. Darrell, T., Gordon, G., Harville, M., and Woodfill, J. (2000). Integrated Person Tracking Using Stereo, Color, and Pattern Detection. International Journal of Computer Vision, 37(2): Demsar, J., Zupan, B., Leban, G., and Curk, T. (2004). Orange: From Ex- Harville, M. (2005). Stereo person tracking with short and long term plan-view appearance models of shape and color. In IEEE, editor, IEEE Conference on Advanced Video and Signal Based Surveillance, AVSS 2005, pages Jain, Heikkilä, A., J. Dass, and S., Silvén, and O. Nandakumar, (2004). A real-time K. (2004a). system Can for monitoring soft biometric of cyclists traits assist user recognition? In Jain, A. K. and Ratha, N. K., editors, Biometric Technology for Human Identification, volume 5404, pages SPIE. Jain, A., Ross, A., and Prabhakar, S. (2004b). An introduction to biometric recognition. Circuits and Systems for Video Technology, IEEE Transactions on, 14(1):4 20.

35 References Horprasert, T., Harwood, D., and Davis, L. S. (1999). A statistical approach for real-time robust background subtraction and shadow detection. In Proc. IEEE ICCV, volume 99, pages Lee, M., Forlizzi, J., Rybski, P., Crabbe, F., Chung, W., Finkle, J., Glaser, E., and Kiesler, S. (2009). The snackbot: documenting the design of a robot for long-term human-robot interaction. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, pages ACM New York, NY, USA. Zhao, L. and Thorpe, C. E. (2000). Stereo-and neural network-based pedestrian detection. Intelligent Transportation Systems, IEEE Transactions on, 1(3):