Robust Multithreaded Object Tracker through Occlusions for Spatial Augmented Reality

Transcription

1 ETRI Journal, Volume 40, Number 2, Aril, Robust Multithreaded Objet Traker through Olusions for Satial Augmented Reality Ahyun Lee and Insung Jang A satial augmented reality (SAR) system enables a virtual image to be rojeted onto the surfae of a realworld objet and the user to intuitively ontrol the image using a tangible interfae. However, olusions frequently our, suh as a sudden hange in the lighting environment or the generation of obstales. We roose a robust objet traker based on a multithreaded system, whih an trak an objet robustly through olusions. Our multithreaded traker is divided into two threads: the detetion thread detets distintive features in a frame-to-frame manner, and the traking thread traks features eriodially using an otial-flow-based traking method. Consequently, although the seed of the detetion thread is onsiderably slow, we ahieve realtime erformane owing to the multithreaded onfiguration. Moreover, the roosed outlier filtering automatially udates a random samle onsensus distane threshold for eliminating outliers aording to environmental hanges. Exerimental results show that our aroah traks an objet robustly in realtime in an SAR environment where there are frequent olusions ourring from augmented rojetion images. Keywords: Augmented reality, RANSAC, Satial augmented reality, Stithing, Traking. Manusrit reeived Aug. 7, 2017; revised Nov. 13, 2017; aeted Nov. 23, Ahyun Lee (orresonding author, ahyun@etri.re.kr) and Insung Jang (e4dol2@etri.re.kr) are with the Hyer-onneted Communiation Researh Laboratory, ETRI, Daejeon, Re. of Korea. This is an Oen Aess artile distributed under the term of Korea Oen Government Liense (KOGL) Tye 4: Soure Indiation + Commerial Use Prohibition + Change Prohibition (htt:// I. Introdution A satial augmented reality (SAR) tehnique is a rojetor-based form of augmented reality (AR). Using suh tehniques, virtual images an be diretly dislayed on the surfae of a real-world objet, as shown in Fig. 1. Mirage Table [1] by Mirosoft is an interative table based on a rojeted AR tehnique. The amera reognizes and detets real-world objets, and the rojetor rojets virtual information onto a table. A future roboti omuter (FRC) [2] enables the movement of rojetor and amera oses using roboti atuators. This means that an FRC an move rojetion images aording to the requirements of the aliation, suh as having a wide rojetion in a restritive environment or hanging the diretion of rojetion. When a rojetor and amera are moved, the alulation of their ose is essential for rendering a rojetion image. This system uses roboti kinematis based on a omuter-aided design or reognizes a marker for ose estimation. Another feature of the SAR tehnique is that a user an use tangible objets as an interfae devie. Figure 1( and d) show examles of SAR systems using tangible objets. Lego Oasis [3] reognizes user-reated Lego bloks and rojets animated rojetion images. A urve design game [4] uses yellow aer markers that an be easily rodued for designing a urve. These systems rovide the user with intuitive ontrol using a tangible interfae by rojeting a virtual image onto the surfae of a real-world objet or onto a worklae. They an rovide an easy and simle user interfae environment for hildren and the elderly. However, SAR systems are unfavorable for deteting or traking realworld objets owing to olusions. Olusions refer to sudden hanges in the lighting environment or the generation of obstales between the amera and the traked objet. They are a serious issue in htts://doi.org/ /etrij ISSN: , eissn:

2 Ahyun Lee and Insung Jang 247 (a) () Fig. 1. Examles of SAR systems: (a) MirageTable [1], (b) a future roboti omuter [2], () Lego Oasis [3], and (d) a urve design game [4]. objet traking when alulating a amera ose in an SAR environment. There are well-known solutions to olusions, suh as a sale-invariant feature transform () [5], a seeded-u robust feature () [6], oriented features from aelerated segment test, and rotated BRIEF () [7], whih are rotation-invariant and resistant to noise. However, these traking methods inur signifiant omutational osts for ahieving realtime erformane. Hybrid feature traking (HFT) [8] with an has reliable traking erformane in a real-time system. Although a multithreaded system detets the features in another thread, it ahieves real-time erformane for alulating a amera osition in the main thread. However, its traking reliability is low in an SAR environment where there are frequent olusions from augmented rojetion images. Simultaneous loalization and maing (SLAM) is well-known for estimating the amera trajetory while reonstruting the environment [9]. It detets a large number of distintive features in the environment and uses bundle adjustment to inrease the auray of amera loalizations. However, we use a single amera mounted on a robot arm. Camera loalizations are estimated by data-driven robot kinemati ontrol [10]. We only use the roosed method for traking a movable objet. Therefore, the number of features from an objet is smaller than that of SLAM aroahes. This means that objet features an be easily affeted by frequent olusions in the rojetion-based AR system. Similarly, Wang [11] uses SLAM for amera loalizations and traks a movable objet using an RGB-D amera. (b) (d) SLAM methods use a random samle onsensus (RANSAC) [12] to eliminate the errors from deteted and mathed features [9, 13]. The RANSAC distane threshold must be redetermined by the user. In addition, these aers do not ontain the ontents of the RANSAC arameters. Cheng [14] resented an automati way to determine the RANSAC threshold based on the rojetion error of the RANSAC. We roosed a traking method that determines a aseseifi RANSAC distane threshold based on an otial flow traker [15]. It enables the threshold to be determined aording to the objet shae and a sale. However, the threshold also must be redetermined manually by the user. In this aer, we desribe a multithreaded objet traker with outlier filtering, as shown in Fig. 2. It enables the RANSAC distane threshold to be determined aording to environmental hanges. In addition, the RANSAC threshold an be udated automatially at user-defined intervals in real-time. The rest of this aer is strutured as follows. In Setion II, we desribe an SAR system model and how to render a rojetion image based on the ose of the rojetor. In Setion III, we desribe the roosed multithreaded objet traker with outlier filtering in detail. In Setion IV, we evaluate the erformane of the roosed traker and show our exeriment results. Finally, in Setion V, we rovide some onluding remarks. (a) () Fig. 2. Traking examle of the roosed objet traker. The yellow oints indiate the traked features in the amera oordinates: (a) augmented rojetion images on the image, (b) traking result of (a), () ourrene of olusions by a hand, and (d) errors removed through outlier filtering. (b) (d)

3 248 ETRI Journal, Vol. 40, No. 2, Aril 2018 II. SAR System Model The SAR system imlemented in this study was tested using an FRC develoed by ETRI [16]. The FRC onsists of five atuators and two rojetor amera unit airs. The amera reognizes real-world objets, and the rojetor rojets additional virtual information onto the surfaes of real-world objets. The amera and rojetor are hysially fixed to eah other. Therefore, if a rojetor amera unit is alibrated in advane, the rojetor ose an be alulated by simly estimating the amera s extrinsi arameters. In this setion, we desribe the SAR system model and the alibration method for a rojetor amera unit. 1. Projetor Camera Model The SAR oordinate system is based on a inhole amera model and must be defined based on the geometri relationshi of the rojetor and through amera alibration [16], [17]. Eah transform homograhy matrix onsists of an intrinsi arameter and an extrinsi arameter. The intrinsi arameter M is a393 matrix. Here, a and b are the sale fators of the u and v axes of the image, u 0 and v 0 are the rinial oints of the oordinate system, and is the skew arameter. 2 3 a u 0 M ¼ 4 0 b v ; (1) X w ¼ R t 0 1 : (2) The extrinsi arameter X w is a matrix. It is omosed of the translation vetor t and rotation matrix R of the x, y, and z axes. In our system, the amera is alibrated using Zhang s method [18], [19]. Here, is a oint in the alibrated amera oordinates, and w is the orresonding oint in the world oordinates, as shown in Fig. 3. In this ase, the relationshi between them is defined as the amera homograhy matrix H w, in whih the intrinsi amera arameter M and extrinsi amera arameter X w are ombined. ¼ M X w w ¼ H w w : (3) In the ase of rojetor alibration, we annot use Zhang s method, whih requires knowing the image resolution in advane. It is diffiult to know the resolution of the rojeted image from a rojetor in advane owing Projetor oordinates World oordinates H w w H to the offset lens of the rojetor [16]. Thus, the rojetor in our system is alibrated using Tsai s method [16], [20]. Here, is a oint in the alibrated rojetor oordinates, and w is the orresonding oint in the world oordinates, as shown in Fig. 3. In this ase, the relationshi between them is defined as the rojetor homograhy matrix H w, in whih the intrinsi rojetor arameter M and extrinsi rojetor arameter X w are ombined. ¼ M X w w ¼ H w w : (4) The relationshi matrix X between the rojetor and the amera is alulated using the inverse matrix of the extrinsi amera arameter and extrinsi rojetor arameter. X ¼ X w X 1 w ¼ X wx w ¼ X 1 : (5) The rojetor homograhy matrix H w is alulated using the extrinsi amera arameter, reomuted intrinsi rojetor arameter, and relationshi matrix X between the rojetor and the amera, whih is alulated using (5). M X X w ¼ M X w ¼ H w ¼ H 1 w : (6) If is a oint in the amera oordinates, and is is the orresonding oint in the rojetor oordinates, the relationshi between them is defined as the matrix H.In the real-time roess, H is alulated using the amera homograhy and the inverse matrix of the rojetor homograhy: ¼ H w H 1 w ¼ H 1 w H w ¼ H : (7) In our imlemented system, the amera and rojetor are hysially fixed to eah other. Therefore, the relation H w Fig. 3. SAR oordinate system. Camera oordinates htts://doi.org/ /etrij

4 Ahyun Lee and Insung Jang 249 matrix H between the rojetor and the amera is a fixed value in an SAR system. 2. Rendering a Projetion Image w In our SAR system, the amera detets a real-world objet, and the rojetor rojets a virtual image onto the surfae of the objet. The rojetor amera unit is alibrated based on the rojetor amera model desribed in the revious setion. First, we need to alulate the extrinsi amera arameter X w to generate a virtual rojetion image. The ersetive-noint (PnP) estimation using RANSAC between the w features deteted in the world oordinates from the objet image and the features deteted in the amera oordinates is alied to alulate X w. Features in the world oordinates must be three-dimensional oints for PnP estimation. All z-axis values of w are zero owing to the use of a lanar objet as the real-world objet. The entral origin oint ð0; 0Þ in the objet image is the entral origin oint (0, 0, 0) of the world oordinates. Finally, the rojetion image I w in the world oordinates is transformed into the rojetion image I in the rojetor oordinates. I ¼ M X X w I w ¼ H w I w : (8) III. Multithreaded Traking System The ose estimation of a rojetor amera unit is the most essential art of an SAR system. It is arried out as PnP estimation alied between the sene features deteted in an objet of the amera inut image and the objet features deteted in the original objet of the world oordinates. The objet features are deteted using an. The sene features are deteted and traked by the roosed multithreaded traker during the real-time roess. Our multithreaded traker is imlemented based on HFT [8] and is divided into two threads: a detetion thread for deteting distintive features and a traking thread for traking features using a yramidal Luas Kanade traker (LKT), whih is an otial-flow-based traker [21], [22]. Figure 4 shows the task flow of the multithreaded objet traker system. The traking thread is alied frame-to-frame for traking the deteted features, whereas the detetion thread is alied eriodially for deteting newly added features as the sene features in a amera inut image. Owing to olusions, traked features are easily removed or deformed through the otial-flow-based LKT. Therefore, the roosed outlier filtering is imerative for removing errors for robust amera ose estimation. 1. Traking Thread w Yes Fig. 4. Work flow of an SAR system using the roosed multithreaded objet traker. The traking thread is the main thread in an SAR system. Therefore, the real-time traking thread runs at the same interval as the amera s frame rate, whih is indiated with the white bloks shown in Fig. 4. Table 1 summarizes the seudoode for the traking thread. When the system starts, we set the objet that will be deteted and traked in the loo roess. We detet features as the objet features from the original objet in the world oordinates. Beause the original image is a twodimensional lanar objet, all z-axis values of the deteted objet features are set to zero. After the loo roess in Table 1 starts, the traking thread traks the deteted sene features in a amera inut image. Aording to the traking results, the extrinsi amera arameter X w is alulated through PnP estimation using RANSAC. The extrinsi rojetor arameter that is hysially fixed to the amera an be

5 250 ETRI Journal, Vol. 40, No. 2, Aril 2018 Table 1. Pseudoode of the traking thread. FUNCTION: MAIN() Detet objet feature w in an objet image by an. Detet sene feature in a amera inut image Set to LKT features 0 i; k ¼ 0 BEGINLOOP: Cature a amera inut image. Trak i by an LKT; its traked result is iþ1. Adative RANSAC threshold deision roess. k = k + 1 IF: k = 100, Udate RANSAC distane threshold. k = 0. IF: Detetion thread is over, Calulate transformed ose of deteted features. Add newly deteted sene features to iþ1. Outlier filtering with w and iþ1 ; its result is iþ1. IF: number of iþ1 is smaller than the user-defined number OR user-defined interval, Detetion thread starts. PnP estimation with w and iþ1 Rendering a rojetion image. i = i + 1 ENDLOOP estimated using X w and X with (6). Here, X is reomuted during the offline roess. Finally, a rojetion image rojeted by the rojetor is visualized. Our SAR system inludes outlier filtering for robust traking through environmental hanges suh as the distane between the amera and the objet, the objet shae, or olusions. The roosed outlier filtering uses a ase-seifi threshold that is automatially udated using the roosed adative RANSAC threshold deision. A. Outlier Filtering The roosed outlier filtering is alied in a frame-byframe manner in the traking thread before PnP estimation. It hooses good features as inliers for traking with an LKT and eliminates unstable features as outliers that are affeted by olusions. Our aroah lassifies them into inliers and outliers based on the relationshi between the objet image and the sene image, as shown in Fig. 5(a and b). The objet image is the original image of the real-world objet, and the origin of the objet image is set as the origin of the world oordinates in Fig. 5(a). The sene image is atured in the loo roess shown in Fig. 5(b). Table 2 summarizes the detailed stes of the roosed outlier filtering. If a transformed feature 0 in the sene image is laed out of the ase-seifi threshold range, it w 0 w 2 w 4 (a) w 3 w 1 0 w Threshold () is regarded as an outlier, as shown in Fig. 5(). Therefore, this imlies that the distane threshold is the rimary fator for lassifying inliers and outliers. We desribe the roosed adative RANSAC threshold deision in the next setion. B. Adative RANSAC Threshold Deision A distane threshold for outlier filtering is determined in real-time based on the adative RANSAC threshold deision. The roosed aroah determines the distane threshold aording to environmental hanges suh as the objet shae and the distane sale between the amera and the objet. Therefore, unlike the revious setting of the distane threshold offline by a user, it enables ontinuous udating of the distane threshold in real-time. As a result, the stability of traking an be imroved. Table 3 summarizes the stes of the adative RANSAC threshold deision. Prior to the loo roess, the (b) 1 3 Outliers Inliers Fig. 5. Deteted and traked features in the objet and sene image and the definition of outliers: (a) deteted features through an in the objet image, (b) results traked by an LKT from the sene image in the loo roess, and () an outlier feature if a transformed feature 0 is out of range. Table 2. Outlier filtering stes for lassifying inliers and outliers. 1) Comute a ersetive rojetion matrix H from between the features w in the objet image and in the sene image using RANSAC. 2) Comute the Eulidean distane values d i between the ith feature i and a transformed feature 0 i ¼ H i. 3) If d i is smaller than a ase-seifi threshold, it is an inlier. In other ases, it is an outlier. htts://doi.org/ /etrij

6 Ahyun Lee and Insung Jang 251 Table 3. Adative RANSAC distane threshold deision stes. 1) Set the initial threshold to t 0 ¼ s=100, where the sale s is the diagonal length of the objet in the amera inut image. 2) The Eulidean distanes d i,j between i;j and 0 i;j are omuted with the jth feature at every ith frame until the nth frame. 3) At the nth frame, d jmean ¼ 1 P n n i¼1 d i;j is alulated as the sum of eah jth feature with all ith frames. 4) If d jmean is twie d mean ¼ 1 P k k j¼1 d j, d mean j mean is removed to exlude imulse errors. k is the number of features. 5) Udate the distane threshold t 1 through the d jmean dataset omuted using Otsu threshold deision [27] methods. Finally, the normalized threshold t nor = t 1 /s is omuted. 6) At every nth frame interval, stes 2) through 5) are reeated, and the normalized distane threshold is udated. Fig. 6. Flowhart of the detetion thread for adding new features to a traked feature dataset in the traking thread. temorary initial threshold is determined when the initial deteted features are added to the traking thread. The initial threshold t 0 is set aording to the objet sale. We set the interval of the threshold deision to 100 frames in our exeriment. The adative threshold deision is erformed every 100 frames in real-time. Here, t nor 9 s is really used as the distane threshold in the outlier filtering. This means that the threshold is adatively hanged aording to the distane between the amera and the objet. 2. Detetion Thread The detetion thread is another thread of the roosed multithreaded objet traker. This thread detets features in the sene image from the amera inut image. However, the detetion method is too slow and is not suitable for a real-time system. In our aroah, detetion does not affet the traking seed of the traking thread beause the detetion thread is generated intermittently under ertain onditions in a new thread. Figure 6 shows a flowhart of the detetion thread when adding new features to the traked feature dataset in the traking thread. When the detetion thread starts at the lth frame, detetion detets new features in the lth frame. The seed of the detetion roedure is generally 340 ms for deteting and mathing about 700 features. For aliation in a real-time system, all roedures must be erformed within 33.3 ms using a 30-fs amera. Thus, newly deteted features annot be added to the lth frame in real-time. Newly deteted features must be added to the next few frames later. New features are added to the traked feature dataset in the traking thread at the kth frame, as shown in Fig. 6. New features are added to kþ1, whih is desribed in Table 1. It annot be assumed that all lth and kth frames are atured from the same amera osition. Therefore, we alulate the ersetive rojetion matrix between the traked features l at the lth frame and kþ1 at the kth frame. kþ1 ¼ H: l : (9) Finally, the transformed ose 0 new of the newly deteted features are omuted using H. 0 new ¼ H:P new (10) when 0 new is added to the traked features kþ1, we need to onfirm that the same features exist between them. If any features of 0 new are the same as one of kþ1, a feature annot be added. We alulate the Eulidean distanes between one of the 0 new features and all of kþ1.ifany distane is smaller than a ase-seifi threshold, we determine that the oint is a duliate. IV. Exeriments 1. Traking Stability and Seed We evaluated the traking exeriment using a rojetor and amera unit for an SAR system through olusions. An olusion is referred to as a sudden hange in the lighting environment or the generation of an obstale between the amera and traked objet. The exeriment was erformed using a deskto omuter with an Intel â 2.67-GHz Core TM i7 CPU, a rojetor (Otoma P320 with a resolution of 1, ), and a USB 2.0 amera (Logiteh C920 with a resolution of ). We omared the traking stability and seed of the roosed multithreaded objet traker using the,, or based on a single thread. The exerimental video frames were atured using a fixed amera and fixed lanar objet for alulating the root-mean-square error (RMSE). The ground truth of the fixed amera ose, as a

7 252 ETRI Journal, Vol. 40, No. 2, Aril 2018 real amera ose, was omuted by measuring the four orner oints of a retangular lanar objet from the amera inut image. Exerimental amera oses were alulated through the PnP estimation results of eah traker. A amera ose is the 4th olumn vetor t of the extrinsi amera arameter X w and is given by ¼ M X w w ¼ M R ðx; y; z; 1Þ T w : (11) We alulated the RMSEs of the Eulidean distane values between the real amera ose and the amera oses estimated by eah traker. The ideal result is a differene in RMSE of lose to zero. Figure 7 shows the first exerimental results in an SAR environment through frequent olusions. The and roosed method traked the objet more robustly than the or, as shown in Fig. 7(a and b). The roessing seed results are shown in Fig. 7(). The seed results inlude only the detetion, mathing, and traking roess times. The is the slowest aroah among those onsidered. On the other hand, the roosed method is the fastest aroah beause it is based on a multithreaded system. In onlusion, we found that our roosed traker is the most effetive method in terms of the traking stability and seed. In another exeriment, we evaluated the ose hanges along the x, y, and z axes by moving the amera or objet. Although the is the slowest traker, it is the most stable aroah, as shown in Fig. 8. The roosed traker shows similar results as the. In the ase of the RMSE RMSE Milliseonds Proosed method (a) Proosed method (b) Proosed method () Fig. 7. RMSEs of the Eulidean distanes for the,,, and roosed methods and eah roessing seed when the amera and lanar objet are fixed: (a) RMSE between the real amera osition and the estimated amera ositions, (b) magnified grah of (a), and () the roessing seed results. x axis y axis z axis Proosed method Proosed method Proosed method Fig. 8. Traking results for the amera osition using the,,, and roosed traker when the amera or rojetor is moving. htts://doi.org/ /etrij

8 Ahyun Lee and Insung Jang 253 Table 4. Proess times of eah ste in the traking thread (400 frames as an average value). Time er frame (ms) Traking thread Trak sene features with an LKT Adative RANSAC threshold deision Outlier filtering Pose estimation Rendering or, some of the results were not exressed in the grahs with a limited size owing to the resene of overly large errors. Table 4 summarizes the roess times of eah ste in the traking thread of the roosed multithreaded traker. The roess time results were measured when traking more than 300 features during 400 frames. 2. Threshold Method for an Adative Threshold Deision A distane threshold value that lassifies inliers and outliers in outlier filtering is an essential fator of the filtering erformane. A threshold method for determining the threshold value for dividing d jmean into 2 was found from the data olleted from eah jth feature during seifi frame intervals, as resented in Table 3. In this study, we attemted to find an effetive threshold method for our SAR system [23] [27]. The exerimental video frames were atured using a fixed amera and fixed lanar objet for alulating the RMSE between the ground truth of the fixed amera ose and the exerimental amera oses. Deending on the threshold method used, t nor is determined differently, and the traking results are also different. Table 5 summarizes the exeriment results for finding an effetive threshold method for our SAR system. The value of t nor is very small beause it is divided by the diagonal length of the objet in the amera oordinates. These exeriment results indiate that Otsu s method was the most effetive way to determine the adative threshold deision for outlier filtering. However, the best way differed aording to the objets. We thought that a threshold method should be an otional arameter in the roosed outlier filtering. In general, a low threshold results in a low differene error. However, a low t nor auses a low number of inliers. As a result, a small number of inliers generates a oor ose estimation result. In addition, we omared the results from another multithreaded traker [8] without using the roosed outlier filtering. 3. Panorama Image Stithing with Real-Time Preview The roosed traker enables the robust traking of a lanar objet in real-time. Therefore, we evaluated anorama image stithing with a real-time review using our traker. Panorama image stithing is a well-known Table 5. RMSE aording to eah threshold method using various lanar objets. Bold numbers indiate the smallest value in eah objet. Threshold deision method Objets Average RMSE Intermodes [23] Iterative [24] Moments [25] Perentile [26] Otsu [27] Without filtering [8] t nor N/A RMSE t nor N/A RMSE t nor N/A RMSE t nor N/A RMSE t nor N/A RMSE RMSE

9 254 ETRI Journal, Vol. 40, No. 2, Aril 2018 No Yes (a) Yes No Yes (b) Fig. 10. An examle of anorama image stithing: (a) real-time review results and (b) blended results using the blending thread. Fig. 9. Workflow of the multithreaded anorama image stithing with real-time review. method for reating a wide-angle image with limited tools or environments. Previous aroahes have been onerned with the quality of the anorama image stithing result or fully automated systems [28] [30]. These stithing rograms selet the soure images, while users redit the stithing result using a guideline that hels ature a soure image. To reate a view of a userdesired shae, trial-and-error is required. In this aer, we roose a multithreaded anorama image stithing method with a real-time review. Our anorama image stithing detets and traks features in real-time and estimates the ersetive rojetion transformation between the amera inut images using the roosed multithreaded traker. The roosed system is omosed of traking, detetion, and blending arts. The blending thread based on an automati anorami image stithing method [31] is diffiult to exeute in real-time; thus, we onfigured a real-time exeutable multithreaded system, as shown in Fig. 9. The initial frame is the objet image that should be traked and deteted in the loo roess. When a user selets a soure image for anorama image stithing, the initial frame is udated as a blending result through the blending thread. features are deteted from the udated initial frame for the later traking roess. Realtime review rendering is erformed based on the traking result until a user selets an additional soure image. Our ontribution is the imlementation of a user-friendly and real-time-based anorama image stithing system. A real-time review an be shown in real-time aording to the user s ontrols, as shown in Fig. 10(a). This an hel the user easily generate a desired anorama image suh as a large land view or building image without trial-anderror. V. Conlusion We introdued a robust multithreaded objet traker with outlier filtering for an SAR system. Our roosed traker is omosed of two threads, the traking and detetion threads. Only the traking thread, as the main thread, is exeuted in a frame-to-frame manner, and it enables an objet to be traked in real-time. However, the otial-flow-based LKT has unstable traking results owing to olusions, whih easily remove or deform the traked features. Our outlier filtering automatially udates the RANSAC distane threshold aording to environmental hanges. In addition, we onsider the most effetive threshold deision method for outlier filtering. The exeriment results show that our aroah with adative threshold-deision-based outlier filtering enables a real-world objet to be robustly traked for an SAR that has frequent olusions from augmented rojetion images. Moreover, we evaluated a anorama imagestithing system with a real-time review by alying the roosed multithreaded traker. We believe that our aroah will be a highly ratial solution for traking lanar objets in other aliations. Aknowledgements This work was suorted by the Ministry of Land, Infrastruture and Transort (MOLIT), Korea under the Urban Planning & Arhiteture (UPA) researh suort htts://doi.org/ /etrij

10 Ahyun Lee and Insung Jang 255 rogram suervised by the Korea Ageny for Infrastruture Tehnology Advanement (KAIA) (grant 13 Urban Planning & Arhiteture 02). Referenes [1] H. Benko, R. Jota, and A. Wilson, MirageTable: Freehand Interation on a Projeted Augmented Reality Tableto, in Pro. SIGCHI Conf. Human Fators Comut. Syst., Austin, TX, USA, May 2012, [2] J.H. Lee et al., FRC Based Augment Reality for Aiding Cooerative Ativities, in 2013 IEEE RO-MAN, Gyeongju, Re. of Korea, Aug. 2013, [3] R. Ziola, S. Gramurohit, N. Landes, J. Fogarty, and B. Harrison, Examining Interation with General-Purose Objet Reognition in LEGO OASIS, in 2011 IEEE Sym. Vis. Lang. Human-Centri Comut. (VL/HCC), Pittsburgh, PA, USA, Set. 2011, [4] A. Lee, J.D. Suh, and J. Lee, Interative Design of Planar Curves Based on Satial Augmented Reality, in Pro. Comanion Publiation Int. Conf. Intell. User Interfaes Comanion,SantaMonia,CA,USA,Mar.2013, [5] D. Lowe, Distintive Image Features from Sale-Invariant Keyoints, Int. J. Comut. Vis., vol. 60, no. 2, 2004, [6] H. Bay, T. Tuytelaars, and L. Van Gool, Seeded-U Robust Features (), Eur. Conf. Comut. Vis., vol. 3951, 2006, [7] E. Rublee, T. Tuytelaars, and L. Van Gool, : An Effiient Alternative to or, 2011 IEEE Int. Conf. Comut. Vis. (ICCV), Barelona, Sain, Nov. 2011, [8] T. Lee and T. Hollerer, Multithreaded Hybrid Feature Traking for Markerless Augmented Reality, IEEE Trans. Vis. Comut. Grah., vol. 15, no. 3, 2009, [9] A. Davison, I.D. Reid, N.D. Molton, and O. Stasse, MonoSLAM: Real-Time Single Camera SLAM, IEEE Trans. Pattern Anal. Mah. Intell., vol. 29, no. 6, 2007, [10] A. Lee, J.H. Lee, and J. Kim, Data-Driven Kinemati Control for Roboti Satial Augmented Reality System with Loose Kinemati Seifiations, ETRI J., vol. 38, no. 2, Ar. 2016, [11] X. Wang, Z. Yao, and Z. Yang, The Use of Objet Traking in Visual SLAM, in IEEE Int. Conf. Al. Syst. Innovation (ICASI), Saoro, Jaan, May 2017, [12] M.A. Fishler and C.B. Robert, Random Samle Consensus: A Paradigm for Model Fitting with Aliations to Image Analysis and Automated Cartograhy, Commun. ACM, vol. 24, no. 6, 1981, [13] R. Mur-Artal, J.M.M. Montiel, and J.D. Tardos, - SLAM: A Versatile and Aurate Monoular SLAM System, IEEE Trans. Robot., vol. 31, no. 5, 2015, [14] L. Cheng, M. Li, Y. Liu, W. Cai, Y. Chen, and K. Yang, Remote Sensing Image Mathing by Integrating Affine Invariant Feature Extration and RANSAC, Comut. Eletr. Eng., vol. 38, no. 4, 2012, [15] A. Lee and J.-H. Lee, Multi-threaded Traker with Outlier Filtering for Satial Augmented Reality, in Int. Teh. Conf. Ciruits Syst., Comut. Commun. (ITC-CSCC), Seoul, Re. of Korea, July 2015, [16] J.-H. Lee et al., Calibration Issues in FRC: Camera, Projetor, Kinematis Based Hybrid Aroah, in Pro. Ubiquitous Robots Ambient Intell., Daejeon, Re. of Korea, Nov. 2012, [17] J.-H. Lee, An Analyti Solution to Projetor Pose Estimation Problem, ETRI J., vol. 34, no. 6, De. 2012, [18] Z. Zhang, A Flexible New Tehnique for Camera Calibration, IEEE Trans. Pattern Anal. Mah. Intell., vol. 22, no. 11, Nov. 2000, [19] J. Weng, P. Cohen, and M. Herniou, Camera Calibration with Distortion Models and Auray Evaluation, IEEE Trans. Pattern Anal. Mah. Intell., vol. 14, no. 10, 1992, [20] R. Tsai, A Versatile Camera Calibration Tehnique for High-Auray 3D Mahine Vision Metrology Using Offthe-Shelf TV Cameras and Lenses, IEEE J. Robot. Autom., vol. 3, no. 4, 1987, [21] J.-Y. Bouguet, Pyramidal Imlementation of the Affine Luas Kanade Feature Traker Desrition of the Algorithm, Intel Cor., vol. 5, 2001, [22] J.-W. Choi, D. Moon, and H.H. Yoo, Robust Multi-erson Traking for Real-Time Intelligent Video Surveillane, ETRI J., vol. 37, no. 3, June 2015, [23] J.M. Prewitt and M.L. Mendelsohn, The Analysis of Cell Images, Annu. NY Aad. Si, vol. 128, no. 3, 1966, [24] M. Fornasier and H. Rauhut, Iterative Thresholding Algorithms, Al. Comut. Harmon. Anal., vol. 25, no. 2, 2008, [25] W.H. Tsai, Moment-Preserving Thresholding: A New Aroah, Comut. Vis. Grah. Image Proess., vol. 29, no. 3, 1985, [26] W. Doyle, Oerations Useful for Similarity-Invariant Pattern Reognition, J. ACM, vol. 9, no. 2, 1962,

11 256 ETRI Journal, Vol. 40, No. 2, Aril 2018 [27] N. Otsu, A Threshold Seletion Method from Gray-Level Histograms, IEEE Trans. Sys. Man. Cybern., vol. 9, 1975, [28] J.H. Cha, Y.S. Jeon, Y.S. Moon, and S.H. Lee, Seamless and Fast Panorami Image Stithing, in IEEE Int. Conf. Consumer Eletron. (ICCE), Las Vegas, NV, USA, Jan. 2012, [29] F. Zhang and F. Liu, Parallax-Tolerant Image Stithing, in Pro. IEEE Conf. Comut. Vis. Pattern Reogn., Columbus, OH, USA, June 2014, [30] Y. Xiong and K. Pulli, Sequential Image Stithing for Nobile Panoramas, in IEEE Int. Conf. Inf., Commun. Signal Proess. (ICICS), Maau, China, De [31] M. Brown and D.G. Lowe, Automati Panorami Image Stithing Using Invariant Features, Int. J. Comut. Vis., vol. 74, no. 1, 2007, Ahyun Lee reeived his PhD degree in omuter siene from the University of Siene & Tehnology, Daejeon, Re. of Korea. From 2011 to 2012, he was an engineer at LG Eletronis In. Pyeongtaek, Re. of Korea. He has been a researh sientist at ETRI, Daejeon, Re. of Korea sine His researh interests inlude omuter vision, augmented reality, robotis, and satial information. Insung Jang reeived BS and MS degrees in omuter engineering from Pusan National University, Re. of Korea in 1999 and 2001, resetively. Sine 2001, he has been a senior member of the researh staff at ETRI, Daejeon, Re. of Korea, and he is also working toward a PhD degree in omuter engineering from Pusan National University. His main researh areas are latforms for geosensor servie, navigation servie, and loation-based servie. htts://doi.org/ /etrij