Real-time mean-shift based tracker for thermal vision systems

Transcription

1 9 th International Conference on Qantitative InfraRed Thermography Jly -5, 008, Krakow - Poland Real-time mean-shift based tracker for thermal vision systems G. Bieszczad* T. Sosnowski** * Military University of Technology, Faclty of Electronics, Kaliskiego Str., Warsaw, Poland ** Military University of Technology, Institte of Optoelectronics, Kaliskiego Str., Warsaw, Poland Abstract In this article, the problems of object tracking with se of digital image processing techniqes are discssed. Two methods of object tracking are described and compared. Evalation of the algorithms will comply with portable thermal camera application.. Introdction Real-time objects tracking is the critical task in many compter vision applications sch as srveillance, object-based video compression, or driver assistance. Object tracking is a process of finding a chosen object on following frame sing knowledge abot its position in previos frames. The most challenging isses encontered in visal object tracking are clttered backgrond, noise, occlsions and change in appearance of tracked objects. The whole process of object tracking is made with a component called tracker. In typical visal tracker we can distingish two major components. One is responsible for target representation and localization, while the second is responsible for filtering and data association with respect to object dynamics. This article focses on the first component responsible for data representation and localisation. Visal tracker has to cooperate with some other components. To start the tracking procedre, the interesting object shold be detected. Detection algorithm has to point interesting object ot for the tracking algorithm. This component is not always necessary. There are systems where an interesting object is pointed by the system operator. Reslts of tracking are also sed in some reasoning and decision making components. Localization of tracking algorithm in video processing chain is shown in Fig.. Tracking cache Data aqision Object detection Tracking algorithm Reasoning algorithms Fig.. Video processing chain. The object tracking ses techniqes of digital image processing. Digital Image is bilt of elementary objects of certain vales that represent lminosity. These elementary objects are called pixels. The pixel set is oriented in perpendiclar Cartesian coordinate system. An object from or area of interest is represented as a sbset of pixels. In typical visal tracking system, the following images are recorded in fixed periods of time and they are called frames: = f f f ( 0,0) f ( 0,)... f ( 0, N ) (,0) f (, )... f (, N ) f, ().. ( M,0) f ( M, )... f ( M, N ) f - single frame the pictre, f ( m, n) - vale of the pixel in m-th row and n-th colmn, M - pictre height, N - pictre width.. Methods revision Tracking algorithms can be divided in for wide categories:. Gradient-based methods locate target objects in the sbseqent frame by minimizing a cost fnction []..

2 9 th International Conference on Qantitative InfraRed Thermography. Featre-based approaches se featres extracted from image attribtes sch as intensity, color, edges and contors for tracking target objects []. 3. Knowledge-based tracking algorithms se a priori knowledge of target objects sch as shape, object skeleton, skin color models, and silhoette [3]. 4. Learning-based approaches se pattern recognition algorithms to learn the target objects in order to search them in an image seqence [4]. Gradient-based methods like SSD (Sm of Sqared Differences) evalate target transition by finding changes between two conseqent frames. Changes are estimated with gradients in space and time. This method has relatively low comptational complexity bt in practice is reliable when small differences between two frames are assred. There is a need to add some special rotines to make this algorithm immne to occlsions and lmination changes. Knowledge-based methods demand excessive knowledge database abot objects appearance and dynamics. They have high comptational complexity and are sed rather in off-line systems. In retrn, these methods provide high accracy. Learning-based approaches nlike previos method class do not se previosly collected knowledge base becase they learn objects properties on the spot. This method is also combined with a detection algorithm becase there is no provided prior information abot which object shold be tracked. These methods are robst bt the sage in real time systems is limited de to high comptational power demands. One of the most robst algorithms sed in object tracking systems is color-based kernel density estimation, introdced in [5], which belongs to featre-based class. What is important, featre space is not limited to color data. It can employ any other featre space extracted from image with sccess. Localization is obtained by comparing and matching areas of image sing estimated featres. This procedre is sally made sing Mean-Shift mode estimation. This approach gained more attention among research commnity de to its low comptational complexity and robstness to appearance change, and eventally evolved too many variations. Adaptation in this method is simpler than in learning-based methods, becase it employs only model pdate. Choosing a proper method depends on needed application. In offline systems, the most sitable wold be ones employing knowledge-based and learning-based methods. In real-time systems, fast and robst methods with low comptational demands shold be sed. There was a need to find optimal tracking algorithm for portable thermal camera. The architectre of thermal camera is shown in Fig.. Fig.. Thermal camera architectre. Search for tracking method for this application was exected with respect to presmptions that are conseqences of thermal camera architectre. In this application, an algorithm shold meet real-time constrains, shold be easy to implement in general prpose processor or FPGA strctre. That s why in this application learning-based and knowledge-based methods were not pt in consideration. Two methods which present the most promising properties for sch application were tested and compared: Mean-Shift and Sm-of-Sqared- Differences methods. These methods are well known and sccessflly sed in typical visal systems, bt are poorly investigated in systems employing thermal images. In frther part of this article, Mean-Shift and Sm-of- Sqared-Differences methods will be described in details and comparison between these algorithms will be performed.

3 IR SIGNATURE AND RECOGNITION 3. Mean shift algorithm To characterize the object, first a featre space has to be chosen. The reference target model is represented by its Probability Density Fnction (PDF) in the featre space. For example, the reference model can be chosen to be the lminosity PDF of the object. The reference model is associated to the point 0 = (0,0) in the spatial domain - centre of the reference object. The target candidate localized at the point y is represented by its own temperatre PDF. Both probability density fnctions are estimated from m-bin histograms. Histogram is not the best PDF estimation bt it is satisfying in this application, and it can be efficiently calclated in hardware like that proposed in [6]. The target and the model have certain size. What is more, in the practice, pixels placed farer form the centre are affected by occlsions, backgrond interference and other noises. To avoid this ndesirable effect the kernel k(r) is added. In reslt, probability of featre of the object is now calclated with a formla below: n p ( ) = xi y p ( y) = cp k δ b( x i), () i h p (y) - probability of the featre in object centered at the point y b(x i) - fnction associating the pixel x i to appropriate featre (here to histogram bin) k(r) - kernel profile h - bandwidth constant that determinate object size (algorithms area of interest) δ(x) - kronecker delta fnction c p - normalization constant To find a new location of an object in following frame, there is a need to find the most similar target object to the model. Similarity is obtained by the special coefficient, called Bhattacharyya coefficient, which is calclated from Eq. (). ( y) ρ[ p( y), q] = p ( y) m ρ, (3) = q P(y) - probability density fnction of featres in object centred at the point y q - probability density fnction of featres in model object. To accommodate comparisons among varios targets, this distance shold have a metric strctre. To achieve this, a distance between two probability density fnctions is calclated sing Bhattacharyya coefficient from Eq. (3). d y = p y, q. (4) ( ) ρ[ ( ) ] This statistical measre has some desirable properties like: it has metric strctre, has geometric interpretation it is a cosine of angles between two m-dimensional vectors p and q. It ses lminosity as a featre space, therefore its invariant to scale and rotation. Minimizing Eq. (4) is made by maximizing Eq. (3). Searching of new target starts from a position of the target in the previos frame B n, and its neighborhood. To redce algorithms compting complexity, the linearization of Bhattacharyya coefficient with Taylor expression arond the point y 0 was performed: m [ p( y), q] p ( y ) q + p ( y) = 0 q m = p 0 ρ (5) ( y ) It trned ot that this linearization provides great redction in comptational complexity with nnoticeable penalty to algorithms accracy. Localization is obtained by searching for the coordinates of similarity fnction maximm. The search of maximm is done with mean-shift procedre [7]. This iterative procedre moves from the location B n to the most similar location in area of interest placed at B n+. Next, the PDF is calclated for the new location B n+. B n+ = n p i= n p i= Yi B Yi wi g h Yi Bn wi g h n (6)

4 9 th International Conference on Qantitative InfraRed Thermography m q wi = = p g ( r) k ( r) ( ) ( b( Y ) i) A δ. = The localization is repeated ntil correlation between the target and the model is low enogh. This means that target was fond. The more practical ending condition can be adopted. Algorithm can finish tracking when difference between estimation of two following points is smaller then the given threshold. For real-time application, constrain in nmber of searching iteration is needed too. Dring tests it trned ot that nmber of iterations is rarely grater than 4, and limit of 0 iterations does not pt any noticeable impact on algorithms accracy. 4. Sm-of-Sqared-Differences Gradient based methods like Sm-of-Sqared-Differences localize targets by analyzing differences between conseqent frames. Finding target movement is performed by searching minimm of cost fnction in space and time. Cost fnction in this approach is a sm of sqared differences. Sm of sqared differences coefficient is a measre of difference between two fragments of images. Both fragments of images shold have eqal size. For two sqare fragments of size h+ by h+ pixels which are centred at the points (x,y) and (,v) SSD coefficient is eqal: SSD [( Y ( x + i y + j) Y ( + i v j) ) ] n n, + = i, j i, j [ h, h].,, (7) If the searched object was detected at the point (x,y) in previos frame, finding its location in following frame wold mean finding (,v) for which SSD coefficient is the smallest. The point (,v) will be a centre of an object in a new frame fond by the algorithm. f n- (x,y) arg min(ssd)= (,v ) f n (,v ) (,v ) ( 3,v 3 ) SSD SSD SSD 3 Fig. 3. SSD coefficient calclated for varios frame fragments. Tracking reslt is obtained by selecting fragment with the lowest SSD. Tracking for all next frames is made in the same way as described above. The fragment containing the fond object became a model for frther comparison. This approach can lead to some ndesired effects. In case where the object tracking will fail for one frame, the algorithm will forget an object. After that, tracking of this object will not be possible. That is why some special rotines shold be added to make this algorithm immne to partial or fll occlsions, noise and changes of appearance like proposed in [7]. 5. Test Reslts The method was simlated and evalated with specially recorded seqences sing MATLAB software. Seqences were chosen to test algorithms in difficlt conditions. They contain the objects that are partially occlded and they appearance change in time. Seqences were containing the frames with resoltion of 384x88 pixels. Seqences were registered with a thermal camera.

5 IR SIGNATURE AND RECOGNITION Fig. 4. First test seqence. First test seqence was prepared to check algorithms basic properties. It contains 5 frames and shows a hman walking towards left side. This seqence is not affected by serios noise or distractions. A pictre of hman is large and clear. a) b) c) Fig. 5. Second test seqence. Second test seqence contains 7 frames. It shows a hman that walks from right side of scene to the centre of scene, stands for several seconds and walks back to the right. In this seqence, appearance of object is changing a lot becase it is observed from different angles dring its movement.

6 9 th International Conference on Qantitative InfraRed Thermography Fig. 6. Third test seqence. Third test seqence contains 80 frames. In this seqence, a tracked hman is entering a scene from right and walks by the arc. Finally, he is leaving a scene on the right side. In this seqence, a tracked object is changing appearance drastically. A pictre of a hman is scaled over time becase of a perspective effect. What is more, in this seqence, appearance of the object is getting similar to the backgrond, what can test algorithms in a context of efficient detail extracting and adaptation. Fig. 7. Forth test seqence. Forth test seqence contains 40 frames. This seqence was recorded to test tracking algorithms efficiency in operation with the objects that contain only several pixels in the pictre. This is highly reqired for a tracking algorithm to handle the objects containing limited amont of pixels, especially for thermal pictres which are of rather low resoltion. For small objects, all noises and distractions have mch grater impact on objects appearance. Image discretisation effects are also considerable.

7 IR SIGNATURE AND RECOGNITION Fig. 8. Tracking algorithms simlation. Mean-shift tracking reslts marked with red color. Sm-of-Sqared- Differences tracking reslts marked with green color. Simlation showed that both algorithms can track objects, bt different properties of these algorithms are noticeable. MS algorithm made better job for the third seqence only. A trajectory was smoother and better fitted to actal one. In the first and second seqence, both algorithms had similar performance. The real demanding forth test seqence showed that SSD algorithm is mch better in tracking small objects. MS tracker failed to track the object in this seqence jst at the beginning. Problems with tracking small objects with MS method are probably cased by errors in PDF estimation for low amonts of pictre data. General properties of these algorithms can be evalated on a simlation basis. Properties were set together in a table bellow: Table. SSD and MS algorithms comparison. Featre Tracking algorithm SSD MS Speed Smaller Grater Tracking range Comptational power constrained Smaller than tracked object Small objects tracking efficiency Grater Smaller Calclation time Constant Variable Change of appearance immnity Smaller Grater Occlsion immnity Additional procedres needed Partial Rotation invariance Smaller Grater Scale invariance Smaller Grater This simlation showed that Mean-Shift based tracker is not sitable for thermal image systems, especially when it has to track small objects. This evalation showed that porting algorithms which are sccessflly sed in traditional vision systems to the thermal vision systems is not always sccessfl. The better wold be sing another algorithm like Sm-of-Sqared-Differences in this particlar application. REFERENCES [] G. Hager, P. Belhmer, Efficient region tracking with parametric models of geometry and illmination, IEEE Trans. Pattern Anal. Mach. Intell. 0 (0) (998) [] M. Isard, A. Blake, Contor tracking by stochastic propagation of conditional density, in: Eropean Conference on Compter Vision, 996, pp [3] G. Cheng, S. Baker, T. Kanade, Shape-from silhoette of articlated objects and its se for hman body kinematics estimation and motion captre, in: IEEE Conference on Compter Vision and Pattern Recognition, vol., 003, pp [4] O. Williams, A. Blake, R. Cipolla, Sparse Bayesian learning for efficient visal tracking, IEEE Trans. Pattern Anal. Mach. Intell. 7 (8) (005) [5] D. Comanici, V. Ramesh, P. Meer, Real-time tracking of non-rigid objects sing mean-shift, Proceedings of IEEE Conference on Compter Vision and Pattern Recognition, Hilton Head, SC, 000. [6] G. Bieszczad, Hardware implementation of pictre histogram estimation in FPGA, SECON 007 conference, Warsaw, Poland. [7] R. Venkatesh Bab, Patrick Perez, Patrick Bothemy, Robst tracking with motion estimation and local Kernel-based color modeling, Image and Vision Compting 5 (007) 05/6 [8] Rafael C. Gonzalez, Richard E. Woods, Digital Image Processing Prentice Hall, ISBN

8