ARTVision Tracker 2D

Size: px

Start display at page:

Download "ARTVision Tracker 2D"

Darrell Cox
6 years ago
Views:

1 DAQRI ARToolKit 6/Open Source ARTVision Tracker 2D Natural Feature Tracking in ARToolKit6 Dan

ARTVision 2D Background Core texture tracking algorithm for ARToolKit 6.

Broaddus. Component from the development of the Daqri Smart Helmet s VIO system.

Does not require pre-training, although this is optional/advised for larger image

2 ARTVision 2D Background Core texture tracking algorithm for ARToolKit 6. Developed and contributed by DAQRI from our Vision & Sensor Group, led by Chris Broaddus. Component from the development of the Daqri Smart Helmet s VIO system. Can work with image collections containing ~100,000 images, but optimised for ~100. Does not require pre-training, although this is optional/advised for larger image sets. Replaces ARToolKit 5 NFT tracker. Faster and more efficient use of resources than ARToolKit 5.

3 ARTracker 2D Objective-C API Java API JNI C# API P/Invoke ARController C API ARTracker ARTrackableAppearance ARTrackable VideoSource libarvideo ARTracker2D ARTrackable 2D ARTrackerSq uare ARTrackableSquare ARTrackableMultiSq uare ARTrackerInst anton ARTrackableInsta nton libartvision libar libinstanton libarg Android Video Push AVFoundati on Video4Linu x2 Camera calibration database sqlite3 curl glog boost CoreMotion OpenGL/ Key: C++ class External library Internal library Hybrid system Binding

4 ARTVision Tracker2D

5 Recognition/Tracking Pipeline Recogniser can detect many pre-trained images. Tracker is initialised from Recogniser result, (initial homography matrix). Only one Recogniser is required for detecting the presence of many many markers. Tracker is initialised from the Recogniser result. Multi-marker tracking capability. One tracker is required per active marker, up to the maximum number of simultaneous markers to be tracked. Recogniser Video Frame ARTracker2D Is Tracking max markers? No Initialise Yes Tracker Tracker Tracker Homography Result

6 How does this work?

7 Example Feature Detection

8 Alterra Ticket

9 Alterra Postcard 2

10 Alterra Postcard 3

11 Alterra Postcard 4

If we know how two images relate to each other, we can use both images to extract information about them.

to characteristics in the scene that we can recognise easily. We call these characteristics features.

12 What is an Image Feature? In computer vision, usually we need to find matching points between different frames of a scene. If we know how two images relate to each other, we can use both images to extract information about them. (Matching, Tracking, 3D Reconstruction etc ) When we say matching points we are referring, in a general sense, to characteristics in the scene that we can recognise easily. We call these characteristics features. So, what characteristics should a feature have for AR applications? It must be uniquely recognisable. Rotation and Scale invariance. Resistant to photometric variance (brightness) Fast processing for real time operation.

13 Common Feature Types SIFT - Scale Invariant Feature Transform. SURF - Speeded Up Robust Features (ARToolkit 5). }Floating point HAAR - Named after Haar-wavelets by Alfréd Haar FAST - Features from Accelerated Segment Test. BRIEF - Binary Robust Independent Elementary Features. ORB - Orientated FAST and Rotated BRIEF. MSER - Maximally stable extremal regions. KAZE - Kaze, meaning Wind in Japanese. AKAZE - Accelerated KAZE features. FREAK - Fast Retina Keypoint (ARToolkit 6). Corners - Harris (ARToolkit 6), Plessey, Förstner, Shi Tomasi.

14 Recogniser 2D Recogniser Detect FREAK features Match Features No match Fail Detected! Match Found

15 FREAK Features Fast REtinA Keypoints. Retinal sampling pattern, bio-inspired. 45 sampling pairs, 512 bits. Binary feature, not floating point. When there are changes due to blur, FREAK performs worse of all, of the all the binary descriptors. (i.e. not so good for tracking).

16 Getting FREAK-y with I.T.

17 Tracker 2D Tracker Harris Detector Optical Flow Tracked TemplateTracker Tracked Tracking! Not enough inliers Not enough inliers Fail Fail

Harris Corners Corners can be thought of as the intersection of two edges, it represents a point in which the directions of these two edges change.

M = E(x, y) = w(x,y) [I(x+u, y+v) I(x,y)] 2 det(m)= λ1 x λ2 λ1 and λ2 are the eigen values of M Harris s metric R = det(m) k(trace(m)) 2 When R is small, which

18 Harris Corners Corners can be thought of as the intersection of two edges, it represents a point in which the directions of these two edges change. The gradient of the image (in both directions) have a high variation, which can be used to detect the presence of corners. M = E(x, y) = w(x,y) [I(x+u, y+v) I(x,y)] 2 det(m)= λ1 x λ2 λ1 and λ2 are the eigen values of M Harris s metric R = det(m) k(trace(m)) 2 When R is small, which happens when λ1 and λ2 are small, the region is flat. When R < 0, which happens when λ1 >> λ2 or vice versa, the region is edge. When R is large, which happens when λ1 and λ2 are large and λ1 λ2, the region is a corner.

19 Optical Flow Tracking Generate an Image Pyramid by stacking scaled versions of the original image. The flow is estimated at each level by matching the features. The flow is interpolated at each level from coarse to fine. Repeat for each level of the image pyramid.

Image Template Tracking Harris corners are used to generate

that define the region, which is initialised from the video

20 Image Template Tracking Harris corners are used to generate image templates. Homography matrix returned from the recogniser shows where to find the image corners in the image frame Small patches that define the region, which is initialised from the video frame. Templates are matched via a sliding window, where the image intensity difference is minimised.

21 ARTVision Tracker 2D Summary Provides natural feature tracking of complex images. Recognition step works for many images, in the order of ~100,000 unique images. Tracker is initialised from detection step. Video Frame ARTracker2D Is Tracking max markers? One tracker required per currently tracked marker. Tracker is reinitialised with detected image each time it is first detected. No Recogniser Initialise Yes Tracker Tracker Tracker The system can be pre-trained for faster initialisation. Homography Result

22 Database Serialisation The Recogniser is able to be trained with the information to detect many images. This process is the rate limiting step when initialising the ARTVision Tracker. Being able to pre-train this component saves a lot of time, particularly when dealing with many many images. ARToolKit6 includes a tool to serialise a trained recogniser database for deployment on other targets platforms. The Recogniser data is serialised to a file and can be deserialised back into operation. Serialize Recogniser Bytes File Deserialize File Bytes Recogniser

23 Q & A Please feel free to ask any questions! ;) You are welcome to me later Dan.Bell_c@Daqri.com or, join the community at [1] Tola, Engin, Vincent Lepetit, and Pascal Fua. A fast local descriptor for dense matching. Computer Vision and Pattern Recognition, CVPR IEEE Conference on. IEEE, [2] Leutenegger, Stefan, Margarita Chli, and Roland Y. Siegwart. BRISK: Binary robust invariant scalable keypoints. Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, [3] Rublee, Ethan, et al. ORB: an efficient alternative to SIFT or SURF. Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, [4]Viola and Jones, "Rapid object detection using a boosted cascade of simple features", Computer Vision and Pattern Recognition, 2001 [5] Alahi, Alexandre, Raphael Ortiz, and Pierre Vandergheynst. Freak: Fast retina keypoint. Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, [6] Calonder, Michael, et al. Brief: Binary robust independent elementary features. Computer Vision ECCV Springer Berlin Heidelberg, [7] C. Harris and M.J. Stephens. A combined corner and edge detector. In Alvey Vision Conference, pages , [8] Many computer vision example images in this presentation were taken from #ComputerVision #AugmentedReality #Hashtag

Yudistira Pictures; Universitas Brawijaya

Yudistira Pictures; Universitas Brawijaya Evaluation of Feature Detector-Descriptor for Real Object Matching under Various Conditions of Ilumination and Affine Transformation Novanto Yudistira1, Achmad Ridok2, Moch Ali Fauzi3 1) Yudistira Pictures;