Procedia - Social and Behavioral Sciences 97 ( 2013 ) Objects tracking from natural features in mobile augmented reality

Transcription

1 Available online at ScienceDirect Procedia - Social and Behavioral Sciences 97 ( 2013 ) The 9 th International Conference on Cognitive Science Objects tracking from natural features in mobile augmented reality Edmund Ng Giap Weng, Rehman Ullah Khan*, Shahren Ahmad Zaidi Adruce, Oon Yin Bee Faculty of Cognitive Sciences and Human Development, Universiti Malaysia Sarawak Kota Samarahan, Sarawak, Malaysia Abstract Real world objects are recognized by tracking less and tracking based techniques. Mobile augmented reality browsers are tracking less systems, which acquires location data using global positioning system and provide information in the form of maps or web links. Tracking based techniques recognize objects through markers or directly real world objects without markers. Marker based systems actually track the markers not the real objects and therefore, these approaches hides the reality. Markerless (direct real object tracking) systems use client-server architecture. However, these are affected by network latency. The Smartphone is capable to recognize and track real world objects without any server and marker. It can guide the users about their location and also provide information in a convenient way. Therefore, an improved algorithm for tracking real world objects through natural features was formulated. The modified version of speed up robust features (SURF) was used for features extraction from live mobile camera image and recognition. The pose matrix from extracted features was calculated by Homography. The adapted algorithm was tested in a mobile AR-prototype application using iphone. It was found from the results that the formulated algorithm recognized and tracked the real world objects from natural features in speedy, easy and convenient way 2013 The Authors. Published by by Elsevier Ltd. Ltd. Open access under CC BY-NC-ND license. Selection and/or peer-review under under responsibility of the of Universiti the Universiti Malaysia Malaysia Sarawak. Sarawak Keywords: Augmented reality; natural features; marker-less; outdoor; image recognition; tracking 1. Introduction Augmented reality using emerging technologies such as global positioning system (GPS), accelerometer, gyroscope, compass and mobile vision, provides a best opportunity to Smartphone users to explore their surroundings. The real world objects can be recognized by using marker based and marker-less augmented reality systems. Mostly, the previous developers used markers based augmented reality systems. However, those systems actually hide the reality and it was also difficult to keep the markers everywhere. Furthermore, the previous markerless approaches use client-server architecture, which is drastically affected by network latency. The markers based augmented reality was applied in different fields like medical visualization, maintenance and repair, navigation and entertainment. However, the markers are not suitable for outdoor mobile augmented reality because markers hide the reality and need to keep everywhere [1]. Its range is also very limited and end- Marker-less natural features based approach [2], can recognize real world objects, such as sights, buildings, and * Corresponding author. Tel.: ; fax: address: rehmanphdar@gmail.com The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license. Selection and/or peer-review under responsibility of the Universiti Malaysia Sarawak. doi: /j.sbspro

2 754 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) living beings and overcome these limitations. Robust feature descriptors such as SIFT [3], SURF [4], and GLOH [5] are most suitable for applications such as image recognition [6] and image registration [7]. These descriptors are stable under different viewpoints and lighting conditions. These descriptors are ideally suited for searching image databases because they are representing feature points as high-dimensional vectors. However, tracking from natural features is a complex problem and usually performed on a remote server [8] [9] [10]. It is therefore a challenging task to use natural feature based tracking in mobile augmented reality applications without server. However, the mobile phones are very inexpensive, attractive targets for outdoor AR. The improvements in Smartphone capabilities and great potential of computer/mobile vision motivated us to design marker-less natural feature based tracking algorithm for mobile augmented reality Related Work Image processing methods can be used in vision-based tracking to calculate the camera pose relative to real world objects like closed loop systems which correct errors dynamically [11]. This is the most active area of tracking research in computer vision methods. The tracking techniques in computer vision can be divided into two classes: feature-based and model-based [12]. Feature-based methods find a correspondence between 2D image features and their 3D world frame coordinates. To calculate camera pose, 3D coordinates of the features are then projected into the observed 2D image coordinates and distance to their corresponding 2D features is minimized [13]. Marker tracking methods can be used to calculate camera pose in real time from artificial markers. The popular ARToolKit library [14] [15] introduced a method for finding the 3D coordinates of the 4 corners of a square marker and [16] introduced an algorithm for calculating camera pose from known features. The key approach of combining pattern recognition and pose calculation was introduced by [17]. After comparing several leading approaches by [18], no new general marker based systems presented, although some researchers explored tracking from LEDs [19]. Tracking from non-square visual markers was introduced by [20] using ring shaped fiducial markers while [21] proposed circular shaped marker clusters with various parameters, i.e., number of markers, height, and radius with single camera tracking from its topology. Circular 2D bar-coded fiducial system proposed by [19] for vision-inertial tracking. It offered a high information density and sub-pixel accuracy of centroid location. Camera pose can also be determined from natural features, such as points, lines, edges or textures. This research was introduced by [22] presenting a paper at IWAR 98 showing how natural features can be used to extend tracking beyond artificial features. This system was able to dynamically acquire additional natural features after calculating camera pose from known visual features to continuously update the pose calculation. In this way they could provide robust tracking even when the original fiducials were no longer in view. Another tracking technique in computer vision is modelbased tracking. This technique uses a model of features of tracked objects such as a CAD model or 2D template of the object based on the distinguishable features. This model was first presented at ISMAR by [23], who used a visual serving approach adapted from robotics to calculate camera pose from a range of model features (lines, circles, cylinders and spheres). They found that that knowledge about the scene improved tracking robustness and performance by being able to predict hidden movement of the object and reduce the effects of outlier data. Model-based trackers are mostly using edges or lines as a feature for pose calculation. A well known approach is to look for strong gradients in the image around a first estimation of the object pose, without exploring the contours [24]. A CAD model was created by hand for piecewise parametric representation of complex objects such as straight lines, spheres, and cylinders by [23]. A real time model-based tracking approach was proposed by [13] where an adaptive system was adopted to improve the robustness and efficiency. Texture was used by [25] who proposed a textured 3D model-based hybrid tracking system combined with edge information, dynamically determined at runtime by performing edge detection. Edge information and feature points ware combined by [26] which let the tracker handle both textured and un-textured objects, and was more stable and less prone to drift. Likewise, [12] proposed a model-based hybrid monocular vision system, combining edge extraction and texture analysis to obtain a more robust and accurate pose computation. Camera pose estimation is the primary technical challenge of Mobile AR. In order to render virtual objects aligned with the real world, the virtual camera must move and rotate in conjunction with the real camera. The quality of the final results is limited by the accuracy of the virtual camera's position.

3 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) The drawback to treating tracking and recognition as completely separate processes is that recognition can be slow. A long time needs to be applied to the first frame, after which the object may have moved too far to be tracked. This research managed to combine tracking and recognition processes. The already extracted image features are used to calculate homography and then find the model-view matrix. In this way it speeds up the system performance and can work in real time. 2. Methodology The modified version of SURF (Speed up Robust Features) algorithm was used for extraction of natural features and tracking of objects [2]. The projection (pose) matrix was calculated from the extracted features using homography techniques, which contain the coordinates of virtual objects for augmentation purposes Calculation of Pose Matrix Pose matrix was calculated by considering a set of points in the first image of a sequence with homogeneous world coordinates (xi, yi, zi). The world coordinates were linked to a set of points in the screen coordinates to find out the object location on the screen. The association between world and screen coordinates is termed as homography. The homography of coordinates is given in the following matrix. (1) coordinates and that each point on the screen is treated as a ray through the camera center. The actual image position was found by dividing the first and second components by the third. The homography is therefore a simple linear transformation of the rays passing through the camera center and it is actually a combination of rotations, scaling, (2) (3) where defines the -th element of H equations: In matrix form we have (4) (5) (6) where is a 9 element vector containing the elements of H. Therefore, with four non-collinear points, it was solved for all the elements of H as follows:

4 756 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) (7) The solution h is thus the null-space of the 8 9 matrix A. It was solved using singular value decomposition method. However, homography, H, cannot be directly used to augment virtual 3D objects into the image. Since, the Z component from pattern space is assumed to always be zero. The intrinsic parameters of camera can be shown by the following matrix: (8) The offline camera calibration techniques proposed by [27] were used for calculating intrinsic parameters and distortion coefficients. The determined intrinsic parameters were f_x = , f_y = , c_x = and c_y = Similarly, the computed distortion coefficients were , , e-03, e-03 and Homography was calculated from the calculated values of intrinsic parameters and distortion coefficients using open CV [28]. The homography matrix was decomposed in rotation, scaling and translation matrix for translation, rotation and scaling of objects. The pose matrix was formulated by the multiplication of (Scale * Rotation) * Translation. The formulated pose matrix was used for registration between real objects and virtual information Implementation of Proposed Algorithm The proposed algorithm was implemented using Objective C, Open surf library [29] and Open CV 2.2 [28]. It was validated using mobile augmented reality prototype and tested with iphone 4s model. The mobile AR prototype recursively use camera frames to extract features and recognize the real world object. Then, the extracted features were used for calculation of pose matrix using homography. The flow chart of the proposed algorithm is shown in Figure Results and Discussion The proposed algorithm was tested by standard image data set [30]. The standard image data set contains sequences of images which exhibit real geometric and photometric transformations, such as scaling, rotation, illumination and JPEG compression. Each image of the data set was tested by the proposed algorithm to verify the calculation of pose matrix. The results of pose matrix for different images are shown in Figure 2 to 8. Figure 2 demonstrates a 3D model of apple over zoomed grafitty after recognition and calculating central pose. Figure 3 shows 3D model over a normal image of bikes, where as Figure 4 illustrates over a rotated image of boat. Figure 5 exhibits 3D model of apple over a bright image of Leuven and Figure 6 expresses over a blur image of trees. A compressed image of UBC is illustrated in Figure 7, while different views of images are given in Figure 8. The pose matrix of proposed algorithm was validated with standard image data set. It was proved to be efficient and practicable in marker-less mobile augmented reality.

5 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) Mobile Camera Augment camera view Video Frame Calculate Camera Pose No Recognition Process Yes Recogniz Fig. 1, Camera recognition cycle. Fig. 2. 3D model of apple over zoomed grafitty. Fig. 3. 3D model of apple over a normal image of bikes.

6 758 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) Fig. 4. 3D model of apple over a rotated image of boat. Fig. 5. 3D model of apple over a bright image of Leuven. Fig. 6. 3D model of apple over a blur image of trees.

7 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) Fig. 7. 3D model of apple over a compressed image of UBC. Fig. 8. 3D model of apple over a changed view image of wall. 4. Conclusion An improved version of the SURF algorithm was used for features extraction from live mobile camera image and recognition of real world objects. Homography techniques were used to determine the pose matrix from extracted features. Different characteristics of virtual objects such as rotation, scaling and translation were controlled by calculated pose matrix. The adapted algorithms were tested in a mobile AR-prototype application using iphone with standard image data set. The proposed algorithm was able to calculate the central pose and display a 3D model over each image from standard data set. The adapted algorithm was found to be efficient and practicable in marker-less mobile augmented reality. However, its speed and accuracy can also be improved by replacing SURF algorithm with another computationally efficient algorithm. Acknowledgments This work was supported by University of Malaysia Sarawak (UNIMAS). The authors would like to thank UNIMAS for this support. References [1] Reitmayr, G. and D. Schmalstieg. Location based applications for mobile augmented reality. in 4th Australasian User Interface Conference Adelaide, Australia: Australian Computer Society, Inc.

8 760 Edmund Ng Giap Weng et al. / Procedia - Social and Behavioral Sciences 97 ( 2013 ) [2] Edmund Ng Gaip, W., et al., A framework for Outdoor Mobile Augmented Reality. IJCSI International Journal of Computer Science Issues, ( 2): p [3] Lowe, D.G., Distinctive image features from scale-invariant keypoints. International journal of computer vision, (2): p [4] Bay, H., T. Tuytelaars, and L. Van Gool, Surf: Speeded up robust features. Computer Vision ECCV 2006, 2006: p [5] Mikolajczyk, K. and C. Schmid, A performance evaluation of local descriptors. IEEE transactions on pattern analysis and machine intelligence, ( 10): p [6] Darrell, K.G.a.T. The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features. in ICCV [7] Lowe, M.B.a.D., Automatic Panoramic Image Stitching Using Invariant Features. International Journal of Computer Vision, : p [8] M. Pielot, N.H., C. Nickel, C. Menke, S. Samadi, and S. Boll Evaluation of Camera Phone Based Interaction to Access Information Related to Posters. in Mobile Interaction with the Real World [9] ARCAMA-3D A Context-Aware Augmented Reality Mobile Platform for Environmental Discovery. Web and Wireless Geographical Information Systems, 2012: p [10] Reitmayr, G. and D. Schmalstieg. Data management strategies for mobile augmented reality. in STARS Tokyo, Japan. [11] Bajura, M. and U. Neumann. Dynamic registration correction in augmented-reality systems. in Computer Graphics and Applications. 1995: IEEE. [12] Pressigout, M. and E. Marchand. Hybrid tracking algorithms for planar and non-planar structures subject to illumination changes. in. 2006: IEEE Computer Society. [13] Wuest, H., F. Vial, and D. Strieker. Adaptive line tracking with multiple hypotheses for augmented reality. in. 2005: IEEE. [14] Kato, H. and M. Billinghurst. Marker tracking and hmd calibration for a video-based augmented reality conferencing system. in. 1999: IEEE. [15] Stricker, D., G. Klinker, and D. Reiners. A fast and robust line-based optical tracker for augmented reality applications. in [16] Park, J., B. Jiang, and U. Neumann. Vision-based pose computation: robust and accurate augmented reality tracking. in : IEEE. [17] Rekimoto, J. Matrix: A realtime object identification and registration method for augmented reality. in. 1998: IEEE. [18] Zhang, X., S. Fronz, and N. Navab. Visual marker detection and decoding in AR systems: A comparative study. in. 2002: IEEE Computer Society. [19] Naimark, L. and E. Foxlin. Circular data matrix fiducial system and robust image processing for a wearable vision-inertial self-tracker. in. 2002: IEEE Computer Society. [20] Cho, Y., J. Lee, and U. Neumann. A multi-ring color fiducial system and an intensity-invariant detection method for scalable fiducialtracking augmented reality. in International Workshop on Augmented Reality (IWAR 98) A.K. Peters, Natick, Mass: Citeseer. [21] Vogt, S., et al. Single camera tracking of marker clusters: Multiparameter cluster optimization and experimental verification. in ISMAR. 2002: IEEE. [22] Park, J., S. You, and U. Neumann. Natural feature tracking for extendible robust augmented realities. in [23] Comport, A.I., É. Marchand, and F. Chaumette. A real-time tracker for markerless augmented reality. in. 2003: IEEE Computer Society. [24] Fua, P. and V. Lepetit, Vision based 3D tracking and pose estimation for mixed reality, in Emerging Technologies of Augmented Reality: Interfaces and Design, M.B. M. Haller, B. H. Thomas Eds, Editor. 2005, Idea Group, Hershey. p [25] Reitmayr, G. and T.W. Drummond. Going out: robust model-based tracking for outdoor augmented reality. in. 2006: IEEE. [26] Vacchetti, L., V. Lepetit, and P. Fua. Combining edge and texture information for real-time accurate 3d camera tracking. in ISMAR. 2004: IEEE. [27] kronick, Offline camera calibration for iphone/ipad-- or any camera, really, in Urban Augmented Reality [28] Intel. OpenCV Computer Vision Library [cited August]; Available from: [29] Evans, C. The OpenSURF Computer Vision Library [cited /4/2013]; Available from: [30] Mikolajczyk K., S.C. Affine Covariant Features [cited September]; Available from: