DETECTION OF MOVING OBJECT BY FUSION OF COLOR AND DEPTH INFORMATION

Transcription

1 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION T. T. Zhang,G. P. Zhao and L. J. Lu School of Automaton Nanjng Unversty of Scence and Technology, Prvate bag 2004 Nanjng, Chna Emals: Submtted: ec., 205 Accepted: Jan. 24, 206 Publshed: Mar., 206 Abstract- Movng object detecton based on color nformaton s easly affected by llumnaton changes and shadows n complex scenes. epth nformaton can provde complementary nformaton. In the paper, a novel method s presented by usng color and depth nformaton. Frstly, we mprove the codebook algorthm by fusng the depth nformaton as the fourth channel n the code word. Next, a compensaton factor algorthm s presented to make the edges accurate. So the fnal detecton result can be obtaned by logc operaton. Experments adapt the publc datasets, and expermental results show that the proposed method can successfully cope wth the lmtatons of color-based or depth-based detecton. Index terms: Object detecton, codebook, edges, color nformaton, depth nformaton. 274

2 T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION I. INTROUCTION Wth the development of nformaton technology, there have been wdely applcatons of computer vson such as vdeo survellance, human computer nteracton and vdeo conference[- 2].Extractng movng foreground objects from a vdeo sequence s the frst crtcal step n vdeo analytcs[3-4]. The most wdely used approach of movng object detecton s background subtracton. The fundamental of background subtracton s that the movng objects detecton obtaned from the dfference operaton between the current frame and reference background model, whch s the exstng background model or re-establshed a background model through statstcal modelng. There are many algorthms based on color nformaton n the lterature about background subtracton, such as Mxture of Gaussans (MOG) [5], non-parametrc kernel densty estmaton[6] and Codebook (CB) [7]. However, on the one hand, above color-based algorthms face many challengng problems ncludng the followng: vulnerable to llumnaton changes; shadows cast by movng objects; camouflage(.e., smlar color between movng objects and the background )[8]. On the other hand, MOG faces a problem that the learnng rate whch s one of the mportant parameters to adapt to background changes s dffcult to adjust. If the model adapts too slowly, sudden change to the background cannot be detected n a wde model. For a hgh learnng rate, background model wll absorb foreground pxels whch are movng slowly[9]. Although non-parametrc kernel densty can quckly adapt to changes n the background process, t has hgh requrements for hardware memory due to large amount of calculaton[0]. In recent years, many authors have come up wth some methods that combne color and depth nformaton to detect movng object. In [], a logcal or s used to combne the dfferent foregrounds that respectvely come from grayscale mage and dstance mage. Although the method cope wth problems that there are smlar color and closed dstance between the objects and background, but t fals to overcome the edge nose. In [2], the author developed the background subtracton based on the Gaussan Mxture Models usng color and depth nformaton. It not only solved the lmtaton of color camouflage, but also decreased the depth nose. However, t s dffcult to deal wth complex scenes. 275

3 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 In ths paper, we present a novel background subtracton algorthm whch s referred as CB C&E. Frstly, depth nformaton s not only as the fourth channel on the codebook algorthm, but also s as the further condton when judgng a pxel belongs to the foreground or background, whch s referred as CB C&. Secondly, a compensaton factor algorthm s ntroduced to make the edge of detected objects accurate, and a seres of logcal operaton are used to generate the fnal result. The results show a quanttatve and qualtatve mprovement n the movng object detecton. II. PROPOSE METHO The overvew of the proposed method s presented nfgure. There are two stages of the proposed algorthm. CB C& RGB epth R C& CF F Fgure. Schematc of object detecton algorthm: CF stands for compensaton factor. F stands for fnal foreground mask. In the stage, depth nformaton, whch s as the fourth channel (R, G, B, ) of codebook algorthm, enhances the condtons of background constructon and foreground detecton. In stage 2, the fnal output F s obtaned by a logcal or operator between R C& obtaned n stage and compensaton factor CF. CF that s acqured through a seres of logcal operatons strengthen the edge detecton. The logcal or operaton between R C& and CF s defned as follows: F R C& CF () Among them,... n wth n total number of frames. The followng wll detal the above stages. a. Codebookbased on color and depth (CB C& ) epth-based algorthm has strong robustness on sudden lghtng changes, hghlghted regons and shadows, whch color-based algorthm cannot overcome. Actually, we can only use the depthbased codebook algorthm to remove shadows and hghlghted. However, when the objects are 276

4 T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION closed to the background, the pxels wll be classfed as the background. In fgure2, we can see that there are shadows n color-based algorthm; moreover, foreground objects are mstakenly detected based on depth algorthm due to the close range between the objects and background. Consequently, not only do we use the depth as the fourth channel, but also we fuse the color and depth nformaton to detect foreground. (a) test frame (b) CB Color (c) CB epth Fgure 2. The example of complcated scenaro;(a) s the test frame; (b) s the result of colorbased codebook algorthm; (c) the result of depth-based codebook algorthm The condtons contan not only the color and brghtness dstortons, but also the depth dsparty. The approach, whch determnes whether one-dmensonal depth pxel matches the codeword, s n a smlar way to the brghtness. and mn max respectvely represent the mnmum and maxmum depth values n a codeword. epth dstorton s allowed to vary n a certan range, [ low, ], defned as: max Therefore, each codeword low h max mn max, mn c ncludes not only v R, G, B, ) but also an eght-tuple (2) ( aux I, I,,, f,, p, mn max mn max q. The logcal dsparty functon s defned as follows: true f Vald ( ) ( low h ) dsparty(, mn, max ) (3) false otherwse Compared wth color dstorton of orgnal Codebook, we add depth nformaton as further condtons n CB C&. If the color dstorton s less than the threshold or color dstorton s between and 2 ( 2. 6 ),meanwhle, dsparty (, mn, max ) s true, The color (x ) n CB C& s true. color(x) s defned as follows: 277

5 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 true f colordst( x, cm ) color( x) ( colordst( x, cm ) 2 dsparty(, mn, max )) (4) false otherwse The sgnfcance of the formula 5 s to determne those crtcal pxels belong to the foreground or the background. ue to the nput pxel s close to the threshold, usng based-color algorthm cannot accurately detect t. Therefore, the depth value of that pxel s consdered as further condton. In the end, the algorthm matches the current pxel value wth the approprate codeword accordng to the computed color, brghtness and dsparty dstortons. BGS(x) follows: s defned as BG f color( x) brghtness( I, I BGS ( x) dsparty(, mn, max ) FG otherwse mn, I max ) (5) b. Compensaton factor and fuson In the stage, we have evaluated CB C& that reduces the mpact of that closed dstance between object and background n the resultant segmentaton wthout havng to perform uneven edges. There s a comparson between the test frame of CB C& and the ground truth obtaned by manual segmentaton n Fgure 3. It can be observed n Fgure 3(a) that the edges are not accurate. (a) CB C& (b) the ground truth Fgure 3. The comparson of the test frame n Fgure 2; (a) s the result of the CB C& method; (b) s the ground truth 278

6 T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION In order to make the edges more accurate, Compensaton factor algorthm CF s desgned and ts framework s shown n Fgure 4. RGB RGB GRAY AF TH < M CF epth Fgure 4.Schematc of compensaton factor based on depth nformaton algorthm: S stands for the subtracton mask. AF stands for averagng flter. TH stands for the decson threshold n pxel unt. M stands for depth enhanced mask. There are four man steps n the compensaton factor algorthm. ⑴Averagng flter Frst of all, we convert RGB mages to gray mages. Then, the average gray levels of each pxel, whch s n a neghborhood that pxels are surroundng n a square wndow, take the place of orgnal gray levels of each pxel by usng averagng flter. T ( j, (... n wth n total number of frames) represent the fltered mages that a generc pxel coordnates n ( j,, and M stands for the half-sde of the squared kernel. ⑵Threshold determnaton M M T ( j, GRAY ( s, t) 2 (6) 4M sm t M In ths step, our goal s to obtan a logcal depth-enhanced mask value between T ( j, and GRAY ( j,. M ( j, s defned as follows: M by comparng the ntensty for GRAY ( j, T ( j, TH M ( j, (7) 0 otherwse A logcal depth-enhanced mask M, whch s a crteron of choosng pxel n the subtracton mask S to get CF, can be completed by repeatng teratvely above-mentoned step for each pxel. ⑶Obtan subtracton mask To obtan subtracton mask CB - S S depth FG frames, whch are bnary mages. S, we perform a subtracton operaton between color FG frames and 279

7 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 S Our purpose, whch s to obtan mssng edge for the orgnal mask R ECB, leads us to consder only postve pxel value n S. epth RGB (8) CB CB S f S ( j, 0 S ( j, j... J; k... K (9) 0 f S ( j, 0 In Equaton (9), ( j, are the coordnates of a generc pxel, J and K stand for the total number of rows and columns respectvely. The approach s specally amed to make a logcal mask contan regons of epth CB wthout ⑷Two logcal and operaton RGB CB. If a pxel comes from a unform color regon, we consder t vald. In that way, ts level of ntensty s really smlar to the medum level of ntensty computed n the surroundng averagng wndow. CF s defned as follows: CF M S (0) III. EXPERIMENTAL RESULTS AN ANALYSIS Fve dfferent approaches have been studed and evaluated wth the publcly dataset. These approaches are the followng ones: the orgnal color-based Codebook(CB Color ), the Codebook based only on depth(cb epth ), the 4 Codebook(CB4) whch the depth nformaton s only as the fourth channel n codeword wthout bas over color threshold, the Codebook based on color and depth(cb C& ), and the proposed method(cb C&E ). Experments based on dataset compare the proposed algorthm wth the other four methods n three dfferent scenes, such as the stuaton that object gradually keeps away from the background, the stuaton that color of object s the same as the background and the stuaton of sudden llumnaton changes. On the one hand, the use of a dataset wth ground truth segmentaton s requred to perform a quanttatve analyss n addton to the qualtatve one; on the other hand, we conducted a quanttatve analyss of PR value. Smulaton of experments s fnshed n VS200 envronment, whle takng advantage of the OpenCV lbrary to assst mage processng. The test dataset comes from the webste( a. Parameter Settngs 280

8 T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION There s a pluralty of parameters that affect the effectveness of the algorthm n together. In order to acheve good overall performance, we have chosen an optmal set of parameters based on a large number of experments. Accordng to tral and error, we determne the threshold TH (n 2.2 secton) s an emprcal value. Table 4. shows the values of these parameters: Table : Parameters selected for the proposed method Parameter Value Parameter Value 0 TH b. Qualtatve Analyss b. Target and Background Smlar stance Fgure 5 shows the qualtatve results n the Wall sequence of test dataset, whch a person hands a book from near and far, from far and near. We select the 74 th frame, the 93 th frame and the 34 th frame for test respectvely, among them the 74 th frame s nearest from the background and the 93 th s furthest from the background. Frame epth 28

9 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 Ground Truth CB Color CB epth CB4 CB C& CB C&E Fgure.5 Results obtaned from the wall sequence As can be seen from the above, there are large area shadow n the CB Color because of the smlar dstance between object and background. CB epth can solve ths problem, but object cannot be detected n the 74 th frame. CB C& shows good effect when fuse the color nformaton and depth nformaton. The CB C&E reduces the nose of edge and makes edge smoother. So CB C&E gets the best result. 282

10 T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION b. Target and Background Smlar Color In ths secton, we choose the Hallway sequence for qualtatve analyss n the test dataset. As shown n Fgure 6, a man holdng whte box goes through the background scenes. The color of box s smlar to the wall. In order to preferably analyze the results, we only capture the whte box porton of the fgure n ths paper. In Fgure 7, we can fnd the object contaned a black shadow cannot be completely detected n CB Color. Although the detecton result s better n CB epth, whte box s only partally detected. To compare the CB epth and CB C& n Fg.7, the effect of CB C& s obvous. CB C&E makes the edge more clearly. (a) Background (b) 258 th Frame (c) 258 th epth Fgure6. The orgnal mage ()CB Color (2)CB epth (3) CB4 (4)CB C& (5)CB C&E (6)Ground Truth Fgure 7. The result of 258 th b. Sudden Illumnaton Changes 283

11 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 Fgure 8 and Fgure 9 show the qualtatve analyss n the Shelves sequence of test dataset. As shown n Fgure 8, there are changes of exposure when 365 th frame swtch to 366 th frame. In Fgure 9, we can fnd that CB Color cannot adapt to sudden changes n llumnaton. There are a lot of noses n CB4. Most of ths nose s fltered by the CB C& by means of the fuson method. (a) 365 th Frame Fgure 8. The orgnal mage (b) 366 th Frame ()CB Color (2)CB epth (3) CB4 (4)CB C& (5)CB C&E (6)Ground Truth Fgure 9. The result of 366 th c. Quanttatve analyss precson ( P ), recall ( R ) and F are the common evaluaton crtera of object detecton. recall s the true postve; precson s the rato between the number of correctly detected pxels and the total number pxels marked as foreground; F -number s a successful combnaton 284

12 F-number T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION of P and R, to comprehensvely evaluate the performance of the algorthm. They are defned as follows: P TP TP FP () R TP TP FN (2) 2 P F R P R (3) TP (True Postve) s the number of pxels as the movng object s correctly detected.tn (True Negatve) s the number of pxels to be detected as the background. FN (False Postve) s msdentfed as the background pxels. FN (False Negatve) s the number of pxels to be mstaken for the movng object. Ths measure not only offers a trade-off between the ablty of an algorthm to detect foreground and background pxels, but also provdes a general evaluaton of robustness of the algorthm. In general, the value of ths estmator s hgher, the better the performance. In Fgure 0 to Fgure 2, curve shows the F -number of evaluaton frames n dfferent scenes. As we can see, the algorthm performance whch s only based on color nformaton or depth nformaton to detect movng object declne or even fal. The method fuson color and depth nformaton gets better effect n dfferent complex envronment. Of course, late fuson makes the algorthm performance more stable CB Color CB epth CB4 CB C& CB C&E frame number Fgure 0.The scene of smlar dstance 285

13 F-number F-number INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH CB Color CB epth CB4 CB C& CB C&E frame number Fgure.The scene of smlar color CB Color CB epth CB4 CB C& CB C&E frame number Fgure 2. The scene of sudden llumnaton changes IV. CONCLUSIONS The method based on color nformaton cannot cope wth classc ssues such as the smlar color of object and background, sudden llumnaton changes, and shadow nterference. In addton, there wll be a leak due to the closed dstance n the method based on depth nformaton. In ths work, we focus on the combned use of depth and color to reduce the mpact of above problems. On the one hand, depth nformaton s as the fourth channel of the codebook and as the further condton of foreground detecton; on the other hand, the compensaton factor further strengthen the lnk between color and depth nformaton so that the edge s sharper. Multple sets of expermental results show the proposed method has hgher detecton performance n a varety of complex envronments. 286

14 T. T. Zhang, G. P. Zhao and L. J. Lu, ETECTION OF MOVING OBJECT BY FUSION OF COLOR AN EPTH INFORMATION ACKNOWLEGEMENTS Ths Project s supported by the Natonal Natural ScenceFoundaton of Chna (Grant No ). REFERENCES [] X.J.Wang, F.Pan and W.H.Wang, Trackng of movng target based on vdeo moton nuclear algorthm, Internatonal Journal on Smart Sensng and Intellgent Systems,vol. 8, No., 205, pp [2] Y.Q.Wang, Y.Z.Zhang, Object trackng based on machne vson and mproved svd algorthm, Internatonal Journal on Smart Sensng and Intellgent Systems,vol. 8, No., 205, pp [3] H..Yang, C.X.Wang, Performance measurement of photoelectrc detecton and target trackng algorthm, Internatonal Journal on Smart Sensng and Intellgent Systems,vol. 8, No. 3, 205, pp [4] Y.Q.Wang, C.X.Wang, Computer vson-based color mage segmentaton wth mproved Knect clusterng, Internatonal Journal on Smart Sensng and Intellgent Systems,vol. 8, No. 3, 205, pp [5] R.Sngh, B.C.Pal, R.A.Jabr. Statstcal representaton of dstrbuton system loads usng Gaussan mxture model,ieee Trans on Power Systems, vol. 25, No., 200, pp [6] J.Lee, M.Park, An adaptve background subtracton method based on kernel densty estmaton,sensors, 2000, pp [7] K.Km, T.H.Chaldabhongse,.Harwood, Real-tme foreground background segmentaton usng codebook model,real-tme Imagng, vol., No. 3, 2005, pp [8] L.M.Hu,L.L.uan, X..Zhang, Movng object detecton based on the fuson of color and depth nformaton,journal of Electroncs & Informaton Technology, vol. 36, No. 9, 204, pp [9] C.Stauffer, W.E.L.Grmson, Adaptve background mxture modelsfor real-tme trackng,ieee Internatonal Conference oncomputer Vson and Pattern Recognton,Fort Collns, USA, June 999, pp

15 INTERNATIONAL JOURNAL ON SMART SENSING AN INTELLIGENT SYSTEMS VOL. 9, NO., MARCH 206 [0] A.Mttal, N.Paragos, Moton-based background subtractonusng adaptve kernel densty estmaton, IEEE Conference ncomputer Vson and Pattern Recognton,vol. 2, No. 2, 2004, pp [] J.Leens, S.Pérard, O.Barnch, Combnng color, depth,and moton for vdeo segmentaton,lncs, 2009, pp [2] E.Mrante, M.Georgev, A.Gotchey, A fast mage segmentaton algorthm usng color and depth map,ieee 3TV-Conference on the True Vson-Capture, Transmsson and splay of 3 Vdeo, Antalya, Turkey, 20, pp