MASTER: Mobile Autonomous Scientist for Terrestrial and Extra-terrestrial Research

Transcription

1 MASTER: Mobile Autonomous Scientist for Terrestrial and Extra-terrestrial Research Dr. Iain Wallace, Dr. Mark Woods Autonomous & Intelligent Systems Group Wednesday 13 th May 2015

2 What is MASTER? Why do we need it? 2

3 Missing the Obvious? NASA/JPL Retracing its steps 3

4 4 Why are SCISYS doing this work?

5 Science Autonomy Heritage 5

6 6

7 Neuro-Fuzzy, Computer Vision Based Science Classification Used FL flight code 7

8 Fusing Technologies Scene Understanding + Tactical Planning = Closed loop science Autonomously detect what s interesting to scientists Autonomously and safely work out how to gather more data We have a basis for an autonomous scientific robotic apprentice 8

9 9 STOP-PRESS: ProviScout win in FP 7 CREST Autonomous Robotic Scientist Showed that autonomous, science data acquisition/investigation was feasible

10 Full Autonomy Trials in Etna, Iceland, or Tenerife Unusual Material WP4 Interesting Stop WP3 Out Crop WP1 WP2 Unusual Material WP0 10

11 Step Back Need for fundamental approach and review 11

12 Objectives Conceive a generic novelty detection system architecture Test with Planetary Science and EO examples But this is an open research problem Currently several algorithmic choices for detection pipeline steps but which ones work best? MASTER is GSP Early TRL so we: built the prototype architecture implementation and a methodology to evaluate the algorithmic choices System allows you to use alternate algs. for different steps and therefore compare 12

13 MASTER Team Prime Contractor, Req s Definition, System Design Development and Test SCISYS State of the Art Literature Review MRG Uni. of Oxford Planetary Science Dataset Uni. of Leicester Earth Observation Dataset Pixalytics 13

14 Robotic ARM Autonomy and Robotics Group Scope Remote Operations Operators Autonomy Framework Mission Planning Payload Insts. Mission Planning Scene Analysis 3D Data Understanding Data Autonomous Navigation Robotic Platform + low-level software 14

15 The MASTER Problem Can you spot what s unusual in this image? 1 image ~10cm traverse ~2.5s operations

16 The MASTER Problem Can you spot it now? 9 images ~1m traverse ~25s operations and you know what you re looking for.

17 The MASTER Problem Can you spot it now? 25 images ~2.5m traverse ~1 min operations

18 The MASTER Problem Can you spot it now? 100 images ~10m traverse ~5 mins operations

19 The MASTER Problem Miss anything else interesting? 250 images, ~25m traverse, ~15 mins operations How many can be saved/returned? How do you choose?

20 MASTER Objectives A domain independent software architecture for detecting scientific events in images.» This was narrowed to detecting (and defining) novelty. Capturing expert scientist s knowledge. Two test domains» Earth Observation» Planetary Science System output is location and extent of regions of interest. Project output is a detailed analysis of the problem and properties of the presented solution.

21 Classes of Novelty Novel Not Novel Expected This case corresponds to easily classified phenomena seen in training. Unusual Phenomena that can be classified, but an expert has somehow indicated would be unusual. Unexpected The phenomena primarily applied to different contexts in training. Classified Novelty Reasonably probable the classifier has identified the phenomena, but it is a statistical outlier. Unclassified Novelty Where the output is above the threshold to create a new class, this is unclassified novelty.

22 Test Datasets - PS NASA/JPL Photojournal Planetary Image archive

23 Test Datasets PS SAFER Collected during ESA SAFER trial.

24 Test Datasets - Seeker Rover NAVCAM from ESA Seeker trial

25 Test Datasets - EO Processed MERIS Level 1 data

26 System Overview

27 Saliency What s outstanding in this image?» Looking for local interest. Many algorithms evaluated» Complete system can select from a library. Example Saliency Maps Left-to-right: Original image, scale-space saliency map, image-signature saliency map.

28 Saliency Evaluation Conclusions AUC SK Pssafer PS EO Performance varies by domain» Varies by phenomena of interest too. SAFER Dataset performance appears poor, but this is a relative measure not absolute.» Reflects the relatively sparse labelling. Seeker data also exhibited very tight error bounds due to very homogenous data. Operating point selection important, as saliency represents a bound on performance.

29 Classifiers classifier Feature Vector 0.8 Well-proven HOG/Visual Words/Spatial histogram based features» Others also tested, ICF, ACF SVM Classifiers» libsvm implementation» One classifier per class» Trained one versus the rest Parameter selection, then training on complete training set.

30 Summary of Classifier Performance On average, they work most of the time

31 Classification Conclusions Performance on the whole good.» Generality exceeded our expectations. Best performance with homogeneity between test and training.» As seen in SAFER and Seeker data. More training examples improve performance.» 25 samples a sensible lower bound in homogenous data. Classes should be well defined.» Poor relative performance of roi and flare classes.

32 Novelty Detection Simple Novelty» If the most confident classifier is above a threshold, assign a not-novel class according to training frequency. Otherwise Unclassified Novelty. Group Novelty Classification» Similar to the above, but considers also many confident classes (confusion) to be unclassified novelty, and defines classified novelty as a large margin to an uncertain class or many unsure classes.

33 System Evaluation What is novelty, in relation to the training data? How important is the detection of novelty in an image? How important is identifying the location and extent of any novelty? How desirable is it to classify regions of interest that are not novel? What is the appropriate trade-off between detecting novelty, and the cost of false positives? Is the problem one of selection is this image novel? Is the problem one of ranking which is the most novel image?

34 Recall Varying Thresholds Recall Varying Novelty Varying Saliency False-Discovery Rate False-Discovery Rate

35 Labelling Future Possibilities

36 Carnot Discovery - Saliency

37 UKSA Chameleon - Classifiers

38 UKSA Chameleon - Performance average time from command to capture (seconds) CPU consumption (watts) sensor power consumption (hardware) memory usage (component specified, Mb) energy cost over idle overseer (joules, rough) Data product or Test std. dev data product size (Mb) notes idle system na na n/a running overseer, no sensors or data capture, average over 10 minutes na na Mb in 253s - generating map xtion m imu Hz SSC Mode 1 rectified images PAIR Hz hokuyo to PC Hz XB3 rectified images Hz commanded map to occupancy grid (includes dtm) n/a SSC 1 pose 5Hz, PC 0.5hz ROR, Mapp SSC Mode 7 rectified image Hz Xtion QVGA@60Hz PC mode 2, 60hz, SOR filtered XB3 PC Hz Xtion VGA@30Hz PC hz, mode 0, SOR filtered SSC Mode 1 VO estimate Hz still camera on tripod teval SSC mode1, 15Hz XB3 VO estimate Hz commanded SSC Mode 1 PC no filtering Hz (commanded) SSC Mode 7 VO estimate Hz mapping eval eval 1Hz, SSC mode1 for PC 1Hz no f generating map SSC SSC 1Hz PC no filter, Mapping 1Hz generating map SSC SSC 1Hz PC no filter, Mapping 1Hz XB3 PC SOR filter Hz XB3 PC ROR filter Hz SSC Mode 7 PC Hz SSC Mode 1 PC SOR Hz SSC Mode 1 PC ROR Hz SSC Mode 7 PC SOR filter Hz SSC Mode 7 PC ROR filter Hz

39 General Conclusions General classifiers show good performance Visual saliency shows good utility. Evaluation of novelty detection is not simple. Labelling and annotation is a big deal.» Several good datasets created. Future application performance baselines have implications:» Where all images are inspected e.g. labelling» Selecting a few images for downlink» No images are inspected e.g. navcams. Evaluation is slow, execution is fast.

40 We are recruiting for autonomy R&D engineers. Thank you See our rovers in the exhibition area this morning!