NATURAL, INTERACTIVE TRAINING OF SERVICE ROBOTS TO DETECT NOVEL OBJECTS

Transcription

1 MUNICH OCT 2017 NATURAL, INTERACTIVE TRAINING OF SERVICE ROBOTS TO DETECT NOVEL OBJECTS Elisa Maiettini and Dr. Giulia Pasquale Joint work with: Prof. Lorenzo Natale, Prof. Lorenzo Rosasco

2 R1 icub

3 What do we need? Target scenario PLANT To water every friday WINDOW To close at night LIBRARY Contains books To dust every week SOFA Elisa s favorite sofa PLANT To water every monday

4 Step 0: Object Recognition Step 1: Object Detection Window Library Plant Sofa Sofa Plant Sofa Are we done with Object Recognition? The R1 perspective. Giulia Pasquale, GTC 2017, San Jose, CA [ ]

5 Approaches to Object Detection What is Where? Sliding window approach: No object No object No object Grid-based approaches [1][2] : Region-based approaches [3][4][5] : It s a sofa! 1. Slide a window on the image 2. Run a classifier for each window 1. Partition the image with a grid 1. Identify Regions of Interest (RoI) 2. Run a classifier for each grid s cell 2. Run a classifier for each RoI We cannot run a classifier for each possible window! [1] Redmond J. et al, 2016 [2] Liu W. et al, 2016 [3] Girshick R et al., 2014 [5] Shaoqing R et al., 2015 [4] Girshick R et al., 2015 [6] Uijlings J. R. R., 2013

6 Region-based approach: Faster R-CNN [5] Where to look? RPN Region proposals ROI pooling Layer fc6 fc7 Classifier What? Classification scores Where? CNN Predicted Bounding boxes Feature Map For each ROI Bounding box regressor [5] Shaoqing R et al., NIPS, 2015 Modularity = Flexibility!! RPN is faster and more efficient than external methods

7 Object Detection task: Robotic setting Robotics brings new challenges but also more information!? Open-set problem Automatic self-supervision Time coherence Contextual information from sensors (e.g. depth)

8 Object Detection for Robotics: our solution 1. Data acquisition [8] and model training 2. Deployment on R1 Bounding boxes Labels Detection system deployed on R1 thanks to the on board Jetson Tx2 [8] Pasquale et al., Frontiers 2016

9 Object Detection for Robotics: our solution icubworld Taransformations dataset [9] icubworld [9] Pasquale et al. IROS 2016 [

10 Object Detection for Robotics: evaluating models 1. Predictions compared with automatically acquired Ground Truth (Mean Average Precision = 0.71) Scenario 1: same HRI setting 2. Validate results: predictions compared with manual Ground Truth (Mean Average Precision = 0.75) Even better! Scenario 2: different scenes New sequences acquisition and manual annotation Promising results: map floor =0.55, map table =0.66 map shelf =0,53

11 Deployment on R1

12 Object Detection on R1 Train details Performed offline on: GPU: NVIDIA Tesla P100 using: CNN: Zeiler and Fergus network [9] DATASET: icubworld Transformations with: Num images: ~27k 2 2 RPN Train: Iterations: 81k Time: ~40 minutes Detector Train: Iterations: 54k Time: ~53 minutes Total train Time: ~3 hours [9] Zeiler M. D. and Fergus R., CoRR, 2013

13 Object Detection on R1 Deployment details Thanks to NVIDIA Jetson Tx2: fast & easy fully autonomous platform easy systems integration CAFFE & YARP & Python Evaluation of Tensor RT framework Regions per Frame Frame per second 100 ~4 300 ~ ~2

14 Contribution: Future steps: 1. Pipeline to overcome lack of manual annotation for robotic platforms 2. Deployment of detection system on NVIDIA Jetson Tx2 on board of R1 1. Further exploit contextual information to improve precision (e.g. time coherence) 2. Extend the system to open sets towards scene understanding task

15 Giorgio Metta Research Director, icub Facility Lorenzo Natale Principal Investigator, icub Facility Lorenzo Rosasco Team Leader, LCSL Vadim Tikhanoff Technologist Researcher, icub Facility Ugo Pattacini Technologist Researcher, icub Facility Marco Randazzo Senior Technician, icub Facility Alberto Cardellino Junior Technician, icub Facility Tanis Mar Postdoc, icub Facility Raffaello Camoriano Postdoc, LCSL Alessandro Rudi Postdoc, LCSL Carlo Ciliberto Research Associate, UCL IRIS Francesca Odone Assistant Professor, Univ. of Genoa And all the team of the icub Facility and the Laboratory for Computational and Statistical Learning

16 R1 is looking forward to meet you! Please come to see it at Booth number

17 Thank you! Any question? Elisa Maiettini Giulia Pasquale