A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERAS

Transcription

1 INTERNATIONAL JOURNAL OF INFORMATION AND SYSTEMS SCIENCES Volume 1, 3, Number 1, 3, Pages Institute for Sientifi Computing and Information A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERAS IGOR KHARITONENKO, WANQING LI, CHAMINDA WEERASINGHE AND XING ZHANG Abstrat This paper presents a onept, arhiteture and an FPGA-based prototype of a smart image sensor for video ameras to be integrated into Intelligent Video Surveillane Systems. The system utilizes a distributed ooperative arhiteture with appropriate level of intelligene at different levels. The sensor performs automated sene analysis and provides immediate response to suspiious events by optimizing amera apturing parameters. Thus, the sensor turns passive video data olleting and reording systems into ative ollaborators with seurity operators being left to only high-level deision making, while automatially arrying out all monotonous work on ontinuous video monitoring. Key Words, image sensors, intelligent amera, video surveillane, vision systems. 1. Introdution Video surveillane and monitoring systems have beome important omponents in the modern seurity infrastruture. More and more ameras are installed to provide effiient surveillane, but this also requires employing a suffiient number of skilled personnel for monitoring. Aording to [1], the most vulnerable part of video surveillane systems is video monitoring personnel. Most of the time, there are no alarming events and the staff may gradually loose onentration on duty. When alarming events are aptured by the system, the human response is very often delayed and is not optimal. Another problem that redues effiieny of video surveillane systems is a ontradition between the required area of view and the sharpness of the aptured objets. The wider area aptured by a amera, the less resolution it an provide to represent detailed objets. As a result, image quality is usually not suffiient to reognize faial or other important features. One may onlude that the fundamental problem restriting broader utilization of video surveillane systems is aused by the onept when the system is onsidered only as an observation and reording devie, entirely relied on human attention and deision-making. There is a distint trend in video surveillane market towards using intelligent systems, whih are expeted to provide effiient assistane to the operators. The ore of these systems is automati sene analysis, whih an be effiiently implemented using distributed ooperative arhitetures with appropriate level of intelligene at different levels. In this regard, development of tehnologies and eletroni omponents for intelligent video surveillane ameras that an automatially optimize their parameters and extrat ritial information for further proessing beomes an Reeived by the editors and in revised form Deember 21,

2 366 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG important pratial issue. This paper presents a onept, arhiteture and an FPGA-based prototype of a smart image sensor for intelligent video amera that an turn passive surveillane systems into ative ollaborators to support seurity operators for immediate and effiient response to suspiious events. 2. Camera arhiteture The smart image sensor inorporates an automati pan, tilt and zoom (PTZ) funtionality implemented based upon [2] with the parameters deided upon aptured ativity within a speified region of interest (ROI). This provides more detailed visual information in zoomed modes while monitoring the full view at the same time. The automati PTZ adjustments an be instantly made based upon motion ativity, olor and illumination hanges of targeted objets. The blok diagram of the smart sensor is shown in Fig.1. The smart sensor is built upon a 1.3-megapixel SXGA CMOS image sensor with digital output. Suh high image resolution allows displaying Zoom-1 (normal zoom, whole image), Zoom-2 and Zoom-4 without ompromising resolution at the video output. This is ahieved eletronially by ropping a required area of the image sensor just within 40ms. The output video frame from the Color Proessing module is 640x256 per PAL field (640x512 interleaved), or digital YUV format for H.263 video ode, depending on the surveillane system arhiteture. The ROI Module always uses ROI data for monitoring of suspiious events regardless of the urrently displayed area. Moreover, PTZ values are diretly used to optimize Color Proessing Module parameters in suh a way that no speifi saling step is performed before or after the interpolation or olor reonstrution of the video frame. Crystal 48MHz lk Cloks Control Module lk Global Control Module lk, trl Image Sensor SXGA Autofous Module lk addr, CFA data Sensor Interfae addr, Module CFA data ROI Module Buffering Control Module CFA data External SDRAM Color Proessing Module PTZ YUV data PAL video Image Enhane Module YUV data PAL Enoder Module Fig 1. Smart Image Sensor Arhiteture Adaptable image enhanement and noise redution features are implemented on the interpolated YUV olor spae within the Image Enhanement module. These funtions applied only to the luminane hannel (Y), result in signifiant improvement in image quality, espeially while the amera operates in poor lighting onditions.

3 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA PTZ based olor interpolation Color interpolation is performed on the Bayer pattern taking into aount the pan tilt and zoom status of the urrent ROI. This approah an be used both for up and down sampling without performing a speifi saling operation. Many methods are desribed for down saling in a olor filter array (CFA) pattern by sub-sampling [3]-[5]. However, no methods are desribed for both up and down sampling of interpolated tri-olor data taking into onsideration the relative loations of R, G and B omponents in the aptured CFA. This implementation uses a method of weighted bilinear interpolation sheme based on relative distane information from the intended interpolated pixel loation to the original R, G and B omponents in the CFA. The weights are based on the zoom mode and the distane from the intended interpolated pixel loation to the original R, G and B omponents in the CFA. The start addresses for aessing the CFA data are determined by the urrent pan and tilt values; whereas the address inrement rate is determined by the urrent zoom value. Zoom values 1,2 and 4 are implemented in the smart sensor. The method of address inrement together with adaptive weight assignment avoids the usual sale fators assoiated with onventional saling. Suh sale fators are typially floating point numbers and also involve division operations, whih are non-amiable for hardware implementation. The pan (P) and tilt (T) values are also onstrained by the urrent zoom (Z) value as shown in the following equations (1) and (2). 1 (1) 0 P < Z 1 (2) 0 T < Z The proessing CFA data window is always set to be 4 4 regardless of the zoom value. This is also a hardware friendly feature, sine only the most demanding proess determines the hardware resoures. The following sub-setions desribe the alloation of weights for performing weighted bilinear interpolation on the CFA data for reonstruting R, G and B olor data for eah output pixel. 3.1 Zoom-1 mode In the Zoom-1 mode, CFA frame of size 1280 x 1024 is interpolated to produe a full olor frame of size 640 x 512. Therefore, when produing the required 640 (x 3 for R/G/B) values per video line, the olumn address is advaned by 2 eah time, traversing the entire aptured data (i.e values). Due to this jump of 2, the CFA pattern onsidered for interpolation remains the same sine Bayer pattern is periodi after eah two rows and olumns. Assuming square pixels, it is possible to ompute the distane from the interpolated pixel loation to eah weight ategory pixel (see Fig. 2). Let the distanes be d1, d 2, d 3 for weight ategories W1, W2 and W3. If a pixel width is taken as 1 unit, d 1 = 0.707, d 2 = and d 3 = In Zoom-1 mode, the weights are assigned as shown in equation (3) to give higher priority to nearby pixels

4 368 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG (3) 1 W d 4 In this ase, W1=0.95, W2=0.035 and W3=0.015 are obtained after normalization to 1. Therefore, W2 and W3 pixel positions are disregarded in Zoom-1 mode. The interpolated values are given by equations (4-6) for easy hardware implementation. (4) R = X2Y2; (5) G = (X1Y2 + X2Y1)/2; (6) B = X1Y1; R G R G G B G B R G R G Loation of the interpolated value G B G B (a) X0 Y0 X1 Y0 X2 Y0 X3 Y0 W3 W2 W2 W3 X0 Y1 X1 Y1 X2 Y1 X3 Y1 W2 W1 W1 W2 X0 Y2 X1 Y2 X2 Y2 X3 Y2 W2 W1 W1 W2 X0 Y3 X1 Y3 X2 Y3 X3 Y3 W3 W2 W2 W3 (b) Fig 2. CFA window for proessing in Zoomx1 mode: (a) CFA pattern, (b) data positions and () distane based weight distribution 3.2 Zoom-2 mode In the Zoom-2 mode, CFA frame of size 640 x 512 is interpolated to produe a full olor frame of size 640 x 512. Therefore, when produing the required 640 (x 3 for R/G/B) values per video line, the olumn address is advaned by 1 eah time, traversing half of the aptured data (i.e. 640 values) per line. Therefore, the CFA pattern onsidered for interpolation hanges (i.e. toggles) aording to the row and olumn number as shown in Table I. Case 1: Idential to Zoom-1 mode, however, in Zoom-2 mode, the weights are assigned as shown in equation (7). 1 (7) W 2 d In this ase, W1=0.76, W2=0.15 and W3=0.09 are obtained after normalization. Sine ()

5 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA 369 the values of W2 and W3 are still signifiantly smaller than W1, if any olor hannel ontains a single W1 loation, this is taken as the interpolated value. This results in equations (4-6) for Case 1. Case 2: For this ase, the interpolated values are given by, (8) R = X1Y2; (9) G = (X1Y1 + X2Y2)/2; (10) B = X2Y1; Case 3: For this ase, the interpolated values are given by, (11) R = X2Y1; (12) G = (X1Y1 + X2Y2)/2;) (13) B = X1Y2; Case 4: For this ase, the interpolated values are given by, (14) R = X1Y1; (15) G = (X1Y2 + X2Y1)/2; (16) B = X2Y2; Table I. Possible CFA pattern ases for interpolation in zoom-2 mode Case REM[row/2]* REM[ol/2]* Bayer pattern RG/GB GR/BG GB/RG BG/GR * Note that REM[.] means the integer remainder resulting from the omputation 3.3 Zoom-4 mode In the Zoom-4 mode, CFA frame of size 320 x 256 is interpolated to produe a full olor frame of size 640 x 512. Therefore, when produing the required 640 (x 3 for R/G/B) values per video line, the olumn address is advaned by 1 every seond olumn ount, traversing quarter of the aptured data (i.e. 320 values) per line. Therefore, the CFA pattern onsidered for interpolation hanges (i.e. toggles) aording to the row and the olumn number as shown in Table II. Notie that the same CFA data window is now used to interpolate 4 pixel values. If relative distane measures are not used, all the 4 pixels will produe the same result, whih will lead to bloking or jagged edges as seen on many digital zooming shemes. For brevity, only the RG/GB CFA window is onsidered to illustrate the effet of weight assignment. This an be extrapolated to the other 3 types of CFA patterns using the desription on Zoom-2 mode. The interpolated value position indiated within brakets near the CFA pattern is with referene to Fig. 3. With regards to RG/GB pattern, Cases 1, 2, 5 and 6 are onsidered.

6 370 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG Case 1: Idential to Zoom-1 mode, however, in Zoom-4 mode, the weights are assigned as shown in equation (17). (17) W 1 d Table II. Possible CFA pattern ases for interpolation in Zoom-4 mode Case REM[row/2] REM[ol/2] Bayer pattern RG/GB(1) RG/GB(2) GR/BG(1) GR/BG(2) RG/GB(3) RG/GB(4) GR/BG(3) GR/BG(4) GB/RG(1) GB/RG(2) BG/GR(1) BG/GR(2) GB/RG(3) GB/RG(4) BG/GR(3) BG/GR(4) Interpolated Value (3) R G R G G B G B R G R G G B G B Interpolated Value (1) Interpolated Value (2) Interpolated Value (4) Fig 3. One of the CFA windows for proessing in Zoom-4 mode. In this ase, W1=0.56, W2=0.25 and W3=0.19 are obtained after normalization. Now, the weights W2 and W3 are signifiant with regards to W1. Therefore, the interpolated values are given by,

7 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA 371 (18) G = (3*X2Y1 + 3*X1Y2 + X1Y0 + X2Y3)/8; (i.e. W1=0.75, W2=0.25, W3=0.00) (19) R = (5.5*X2Y2 + X2Y *X0Y0 + X0Y2)/8; (i.e. W1=0.69, W2=0.25, W3=0.06) (20) B = (5.5*X1Y1 + X1Y3 + X3Y *X3Y3)/8; (i.e. W1=0.69, W2=0.25, W3=0.06) It an be seen that the hardware amiable equations slightly depart from the omputed weight values; however, this is required to optimize the required hardware resoures. Case 2: Due to the shift in the position of the interpolated pixel, the weight distribution is altered for this ase. The new weight distribution is shown in Fig. 4. W6 W4 W3 W4 W5 W2 W1 W2 W5 W2 W1 W2 W6 W4 W3 W4 Fig 4. Weight distribution for Zoom-4 mode ase 2. Geometri analysis produes the following distane measures for eah weight ategory. d 1 = 0.5, d 2 = 1. 12, d 3 = 1. 5, d 4 = 1. 8, d 5 = and d 6 = 2. 5 Using Equation (9) and subsequent normalization produes the weights, W1=0.4, W2=0.18, W3=0.14, W4=0.11, W5=0.09 and W6=0.08. Therefore, the interpolated values are given by, (21) G = (4*X2Y *X1Y *X3Y2+X2Y3)/8; (i.e. W1=0.5, W2=0.375, W3=0.125) (22) R = (6*X2Y2 + 2*X2Y0)/8; (i.e. W1=0.75, W3=0.25) from W1, W3, W5 and W6 (23) B = (3*X1Y1 + X1Y3 + 3*X3Y1 + X3Y3)/8; (i.e. W2=0.75, W4=0.25) from only W2 and W4 It should be noted that eah olor hannel represents not all the weight ategories. Therefore, it is neessary to re-normalize the weight ategories that are within eah olor hannel disregarding the weight ategories not represented, as in the ase of R and B. Case 5: Due to the shift in the position of the interpolated pixel, the weight distribution is altered for this ase. The new weight distribution is shown in Fig. 5.

8 372 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG W6 W5 W5 W6 W4 W2 W2 W4 W3 W1 W1 W3 W4 W2 W2 W4 Fig 5. Weight distribution for Zoom-4 mode ase 5. Although the weight distribution has been hanged, the distanes to weight ategories and hene the weights are as same as in ase 2. Therefore, the interpolated values are given by, (24) G = (1.5*X2Y1 + 4*X1Y *X2Y3 + X3Y2)/8; (i.e. W1=0.5, W2=0.375, W3=0.125) (25) R = (6*X2Y2 + 2*X0Y2)/8; (i.e. W1=0.75, W3=0.25) from W1, W3, W5 and W6 (26) B = (3*X1Y1 + 3*X1Y3 + X3Y1 + X3Y3)/8; (i.e. W2=0.75, W4=0.25) from only W2 and W4 Case 6: Due to the shift in the position of the interpolated pixel, the weight distribution is altered for this ase. The new weight distribution is shown in Fig. 6. This is a speial ase sine the interpolated pixel is loated on one of the aptured CFA data. Therefore, olor hannel value is generated from this pixel and all other same olor values are disregarded in weight distribution setting. -- W4 -- W4 W4 W3 W2 W3 -- W2 W1 W2 W4 W3 W2 W3 Fig 6. Weight distribution for Zoom-4 mode ase 6. Geometri analysis produes the following distane measures for eah weight ategory. d 1 = 0.0, d 2 = 1. 0, d 3 = 1. 4 and d 4 = 2. 2 Sine d 1 = 0.0 it is not used in the weight omputation. Using Equation (9) and subsequent normalization produes the weights, W2=0.46, W3=0.33, W4=0.21. Therefore, the interpolated values are given by, (27) G = (2*X2Y1 + 2*X1Y2 + 2*X3Y2 + 2*X2Y3)/8;

9 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA 373 (i.e. W2=1.0) from only W2 and W4 (28) R = (8*X2Y2)/8; (i.e. W1=1.0) from only W1 (29) B = (2*X1Y1 + 2*X1Y3 + 2*X3Y1 + 2*X3Y3)/8; (i.e. W3=1.0) from only W3 If the interpolated pixel is loated on the CFA pixel or a partiular olor hannel is signifiantly represented only by a partiular weight ategory, it is neessary to alloate a weight of 1.0 for that partiular ategory, as seen in this ase. This zooming strategy an be used for Zoom-8 and higher zooming levels. In Zoom-8 mode, the weights are assigned as shown in equation (30). 1 (30) W d This method an also be used in onjuntion with gradient detetion, to avoid using pixel values aross edges. However, additional omputations are required in this ase, hene inreasing the hardware resoures needed. 4. Color orretion Color orretion is performed using an algorithm developed by the authors, whih is desribed in detail in [6]. The Green hannel is unaltered. Red and Blue hannels are orreted using the following formulae. (31) R = R R G (32) = B ( B G ) B (33) G = G The R, G, B values represent the average hannel value omputed for eah CFA window of size 4 4. This omputation is performed during the olor interpolation, disregarding the weight values. The resulting values are lipped to be within the range [0 255]. 5. Sharpness enhanement Methods of sharpness enhanement found in literature [7][8] are in the ontext of ompression algorithms (e.g. JPEG), whereas a few deal diretly on the luminane (Y) and hrominane (U/V) signals [9]. The onstrains on the urrent implementation are outlined below: 1. Sharpening and noise redution should be implemented on the YUV olor spae 2. Algorithms should be line based to avoid using temporary storage spae 3. Algorithm filter taps should be minimized to preserve as many output pixels, sine eah extra tap eliminates one valid pixel. Under these onstraints, a simple non-linear sharpening filter that proved its effiieny in [2] was implemented. The filter has the following properties: Filter taps:,,,, Y = S 1 S 1 S 2 S 1 S 0 S + 1 S + 2

10 374 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG Filter oeffiients: If ( Y > 20 ) then {-0.25, -0.25, 2.00, -0.25, -0.25} else {0.00, 0.00, 1.00, 0.00, 0.00}. Although the sharpening is performed only on the Y hannel, it is important to store the orresponding U and V values to avoid olor artifats. This method performs well for natural images to enhane the appearane of sharpness to the human viewer without signifiantly inreasing the noise level. However, the non-linear proessing may inflit some artifats espeially near high frequeny thin lines (above 5MHz). By enhaning the gain of the spatial frequenies in the viinity of the peak response of the Human Visual System (HVS), it is possible to make the output image appear sharper and with higher ontrast. 6. Noise redution The implemented noise suppression algorithm is based on anisotropi diffusion [9]. Anisotropi diffusion is performed along eah video line, for Y hannel only. The main reason for this is to minimize the data buffers needed for storing intermediate data between iterations. Only 3 iterations are performed using intermediate storage. Although only Y hannel is proessed, orresponding U and V values should also be saved, to avoid olor shifts at the output image. Diffusion algorithms remove noise from an image by modifying the image via a partial differential equation (PDE). Anisotropi diffusion stops the diffusion aross edges preserving the edge sharpness in the image [10]. Muh researh has been onduted on the harateristis and behavior of anisotropi diffusion on images [11]. The following desribes the implementation of a line based anisotropi diffusion algorithm for noise redution while preserving the edge sharpness. The orretion weights for eah pixel Y value are based on its immediate horizontal gradient. Consider the following senario: Y, Y, t 2 t 1 Yt The whih replaes the is omputed as follows: Y t Y t 1 (34) 1 = Y t 2 Yt 1 (35) 2 = Y t Yt 1 The orretion weights C are produed as shown in Table III. (36) Y t = Y t 1 + C C 2 2 The resulting Y t value is lipped to be within the range [0 511]. This algorithm is partiularly effetive in high noise onditions (e.g. at low light).

11 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA 375 Table III. Corretion weight based on absolute gradient value 1 C 1 2 C 2 > > Motion detetion and objet traking Motion analysis and objet traking have been studied for several deades. The algorithms of motion detetion over temporal frame differene methods, inluding mean-absolute or sum-of-squares differene, and ross-orrelation tehniques [12], bakground modelling [13] and optial flow methods. The optial flow method is the most aurate, but also it is the most omputationally demanding and is very diffiult to implement in a amera-based proessing unit. The proposed algorithm onsists of five onseutive stages. Its flow hart is shown in Fig.7. First of all, eah frame of the aptured video-sequene is redued in size to bring omputational omplexity to the level suitable for hardware implementation. Based upon a urrent and the previous frame, a ROI with motion ativity is identified. This window is used to optimize pan, tilt and zoom parameters of the image sensor. To improve visual stability of the displayed window with ROI, a method of ballisti smoothing of the pan and tilt oordinates is employed. Fig. 7. A flow hart of the motion detetion and objet traking algorithm

12 376 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG 7.1 Frame size redution To redue the frame aptured with 1280 x 1024 pixel resolution, it is subdivided into a number of bloks 64x64 pixels eah. Therefore there are 320 (20 x 16) bloks B(m,n) alulated aording to (37). k 1 k 1 1 (37) B ( m, n) = f ( km+ x, kn+ y) 2 k x= 0 y= 0 where f(x, y) is the initial high-resolution frame aquired from image sensor and k is the blok size, whih is equal to 64. In other words, the redued size frame is obtained by blok based averaging of the aptured high-resolution frame. Only the redued size frame (20x16) is used then for motion ativity detetion. 7.2 Frame differene alulation The frame differene alulation is based on the weighted differene algorithm, whih takes into aount the blok values from several onseutive frames. Suh temporal filtering redues the influene of inonsistent motion due to the noise. The differene frame has size 20x16 and onsists of the values D t (m,n) alulated aording to (38). (38) D t ( m, n) = 3D t 1 ( m, n) + ( B t ( m, n) B t 1 ( m, n)) Calulation of ROI If the absolute value of D t (m,n) is equal or greater than the threshold T, then the orresponding 64x64 blok is indiated as a motion ativity area. Its alarm flag A t (m,n) is alulated aording to (39). (39) A t ( m, n) = 1 0 D t ( m, n) T otherwise ROI is defined as the one that inludes all bloks, whih have non-zero A t (m,n) aording to (40). (40) l m t r m t t n t b n t = min( m A t ( m,*) > 0) = max( m A t ( m,*) > 0) = min( n A t (*, n) > 0) = max( n A t (*, n) > 0) where supersripts l, r, t, b relate to the left, right, top and bottom blok boundaries of the ROI respetively. The zoom fator Z an be alulated using the following (41).

13 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA 377 (41) Z = 1, 4, 2 r m t r m t l m t l m t otherwise W > 2 W < 4 OR AND b n t b n t t n t > t n t H 2 < H 4 where W is the width and H is the height of bloks within the redued size frame. W=20 and H=16 in our implementation. The position for panning P and tilting T are defined in aordane to (42) and (43) (42) (43) P = T = m t n t W 2 * Z H 2 * Z where l r m t = ( m t + m t ) / 2 is the entral position of the ROI in horizontal diretion and t b n t = ( n t + n t ) / 2 is the orresponding enter in vertial diretion. 7.4 Ballisti smoothing Experiments with the intelligent amera [2] showed that simple appliation of the panning and tilting values, P and T, alulated from equations (42) and (43) for repositioning ROI had resulted in a jittered video sequene, espeially when several parameters have to be hanged at same time. In order to stabilise the sequene, the onept of ballisti smoothing was introdued. The desribed ballisti smoothing algorithm implements a seond-order low-pass temporal filter. Conventional CCTV ameras have ertain mehanial properties (mass, frition, damping, et.), whih affet their operation. Beause there are no physially moving parts in the intelligent amera with the smart image sensor, the mehanial properties of suh pseudo-motion had to be simulated. The smoothing is applied to the entre position of the deteted moving ROI at the original sensor resolution. First, the ROI entre position is onverted from blok window spae to pixel spae using equations (44) and (45). (44) x = m * k ( k / 2) t t + (45) y = n * k ( k / 2) t t + Then, the aeleration of the moving ROI (in pixels per frame 2 ) is alulated as

14 378 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG follows. (46) x a t = 2, 1, 0, x t x t x t 1 x t 1 otherwise > Distmax AND > Dist min AND x s t 1 < Sp MAX x t x t 1 < Distmax AND x s t 1 < Spmax ( 47) y a t = 2, 1, 0, y t y t y t 1 y t 1 otherwise > Distmax AND > Dist min AND y s t 1 < Spmax y t y t 1 < Distmax AND y s t 1 < Sp max where x y x t 1, y t 1 are oordinates of the ROI in the previous frame, s, s are the t 1 t 1 speed of the ROI alulated in the previous frame, Sp max is the maximum permitted speed of the ROI and Dist min, Dist max are minimum and maximum allowable travelling distanes of the ROI between previous and urrent frames. (48) (49) (50) (51) The speed of the ROI (in pixels per frame) is defined as x x x s t = s t 1 ± a t y y s t = s t 1 ± y a t The position of the new ROI is alulated using equations (50) and (51). x t = x t 1 ± y t = y t 1 ± x s t y s t The oordinates of the new ROI together with the zoom fator are sent to the Colour Proessing Module that selets the orresponding area and applies PTZ based olour interpolation. 8. Auto-fousing Auto-fousing algorithm desribed in [14] uses data from ROI to adjust fous. The algorithm was designed to provide stable performane in poor lighting onditions. This feature has proved to be very useful for video surveillane ameras too. Sine the fous-measuring operator of the algorithm uses ROI data, the aptured area is always guaranteed to be in fous. When there is no motion ativity and the sensor automatially swithes into Zoom-1 mode, a user defined ROI is used for fousing.

15 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA Implementation and evaluation Fig.8 shows a prototype of the smart amera. The amera was tested in different surveillane senarios. These senarios inlude both indoor and outdoor senes, suh as monitoring of the airport air traffi from the distane, monitoring of the vehiles at the highway, surveillane at the exhibition pavilion, hallway and lab monitoring. The tests showed that amera produed images with good ontrast and olour fidelity (better than or omparable to middle range CCD surveillane ameras as evaluated by seurity professionals) and had a good response to the motion in the field of view, zooming into the area, whih enloses all moving objets and traking these objets when they move. Fig 8. A Prototype of the intelligent amera The ballisti smoothing proved to be a useful enhanement to the algorithm. It gave a feeling of natural traking and enhaned the stability and omprehension of the video sequenes. Also, it ated as a temporal filter for sudden hanges/motion in the field of view. The hallway traking sequene is presented in Fig.9. The white bars on the right bottom side of the frames show the zoom fator (one bar for zoom 1, or the whole view field, two bars zoom 2 and four bars for zoom 4). The blinking red bar in the top right orner of the frames indiates that the motion was deteted. The first frame shows no movement and the amera stays in whole area view mode (zoom 1). As soon as the person enters the view field (frame 2), the motion is deteted and the amera defines the boundaries of the motion area. Next, amera zooms into the area were motion was deteted (zoom 4), keeping the person who is moving in the entre of the frame. At this zoom the faial features of the person are muh more distinguishable than in the normal zoom (frame 2). As the person moves loser, the motion area grows bigger (frame 4), and as soon as it does not fit into the zoom 4 window, the zoom fator hanges to 2 in order to keep the whole moving objet in the view (frames 5-8). When zoom 2 is not big enough, the amera swithes zoom to 1, the whole area view (frame 9). At last, when there are no moving objets inside the sene (frame 10), the amera automatially turns to the normal zoom to ontinue monitoring the whole view area.

16 380 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG The amera was demonstrated at a number of exhibitions and generated broad interest from the representatives of seurity/surveillane ompanies. (1) (6) (2) (7) (3) (8) (4) (9) (5) (10) Fig 9. Hallway motion traking senario 9. Conlusion The desribed arhiteture was implemented on a one million gates FPGA-based platform. Complexity of the FPGA-based implementation indiated feasibility of integration of the desribed solution with a CMOS image sensor. It was tested in different surveillane senarios. These senarios inluded both indoor and outdoor senes, suh as monitoring of the vehiles at the highway, surveillane at the exhibition pavilion, hallway and lab monitoring. Test onditions, test equipment and the obtained results are desribed in details in [2]. The tests showed that amera resolution was up to 800 lines. It had a quik response to the suspiious ativity, automatially zooming into

17 A PROTOTYPE OF INTELLIGENT VIDEO SURVEILLANCE CAMERA 381 the optimal size ROI, whih enloses all moving objets and traking these objets when they move. It should be pointed out that the amera ontinues monitoring the entire sene while displaying the ROI. If motion ativity is deteted beyond the urrent ROI, the amera an automatially adjust the area size. Thus, the objets are aptured by the amera with the maximum possible resolution that failitates further automati high-level analysis, suh as human ativity identifiation [15][16]. This omputationally expensive proessing an be arried out by the reording/proessing unit of the surveillane system only on those video sequenes, whih ontain motion ativity. In long-term this approah an lead to development of fully automati self-guarded systems. REFERENCES [1] C. Regazzoni, G. Fabri, G. Vernazza. Advaned Video-Base Surveillane Systems. Kluvwer Aademi Publisher [2] C. Weerasinghe, W. Li, M. Nilsson I. Kharitonenko. Digital Zoom Camera with Image Sharpening and Noise Redution. IEEE Transations on Consumer Eletronis, Vol. 50, pp , August 2004 [3] M. Guarnera et al, Method for proessing digital CFA images, partiularly for motion and still imaging, U.S. Patent Appliation No. 2003/ A1 [4] H. Fukuda, Image pikup apparatus, U.S. Patent Appliation No. 2003/ A1 [5] K.A. Parulski, Color filters and proessing alternatives for one-hip ameras, IEEE Trans. on Eletron Devies, Vol. ED-32(8), pp. 1381, 1985 [6] I. Kharitonenko, S. Twelves and C. Weerasinghe, Suppression of noise amplifiation during olor orretion, IEEE Trans. Consumer Eletronis, Vol. 48 (2), pp , 2002 [7] M.Fisher, J.L. Paredes, G.R. Are, Weighted median image sharpeners for the World Wide Web, IEEE Trans. on Image Proessing, Vol. 11 (7), pp , 2002 [8] K. Konstantinides, V. Bhaskaran, G. Beretta, Image sharpening in the JPEG domain, IEEE Trans. on Image Proessing, Vol. 8 (6), pp , 1999 [9] S. J. Huang, Adaptive noise redution and image sharpening for digital video ompression, Pro. of IEEE International Conferene on Systems, Man, and Cybernetis, Vol. 4, pp , Ot., [10] Perona P. and Malik J., Sale-spae and edge detetion using anisotropi diffusion, IEEE Trans. Pattern Anal. Mah. Intell., Vol. 12, No. 7, PP , July [11] Blak M.J. and Marimont D.H., Robust Anisotropi Diffusion, IEEE Trans. On Image Proessing, Vol. 7, No. 3, pp , Marh [12] D.J. Connor and J.O. Limb. Properties of frame-differene signals generated by moving images. IEEE Trans. Communiations, COM-22(10): , [13] Y.W. Huang, B.Y. Hsieh, S.Y. Chien, and L.G. Chen, Simple and effetive algorithm for automati traking of a single objet using a pan-tilt-zoom amera. Pro. of ICME 2002, Switzerland. [14] I. Kharitonenko, X. Zhang. Digital Fous Detetor for Mobile Video Communiators. IEEE Transations on Consumer Eletronis, Vol. 46, pp , February 2000 [15] Wei Niu; Jiao Long; Dan Han; Yuan-Fang Wang, Human ativity detetion and reognition for video surveillane, 2004 IEEE International Conferene on Multimedia and Expo, Vol. 1, pp

18 382 I. KHARITONENKO, W. LI, C. WEERASINGHE AND X. ZHANG [16] N.D. Bird, O. Masoud, N. P. Papanikolopoulos, Detetion of loitering individuals in publi transportation areas, IEEE Transations on Intelligent Transportation Systems, Volume 6, Issue 2, pp , June Igor Kharitonenko was born in Odessa, Ukraine. He reeived the B.S. with honors in eletronis engineering and Ph.D. degree from Odessa Polytehni University in 1985 and 1993 respetively. Dr. Kharitonenko was a prinipal researh engineer at Motorola Australian Researh Centre ( ) working on tehnology development for digital ameras and mobile video ommuniators. He is urrently with University of Wollongong (Australia). His researh interests inlude mahine vision, CMOS image sensor arhitetures, image and video ompression. Wanqing Li reeived his B.S. in physis and eletronis and M.S. in omputer siene from Zhejiang University (Xiqi Campus), China in 1983 and 1987 respetively. In 1997, he reeived a PhD in eletroni engineering from The University of Western Australia. He was a Leturer from 1987 to 1990 and Assoiate Professor from 1991 to 1992, both with the Department of Computer Siene and Tehnology, Zhejiang University of China. From 1992 to 1993, he was a visiting Researh Fellow with the Computer Siene Department, Murdoh University, Australia. From 1997 to 1998, he worked as an Information Tehnology Offier with the Bureau of Meteorology, Australia. He joined Motorola Australian Researh Centre in 1998 as a Senior Researh Engineer and later beame a Prinipal Researh Engineer. Sine 2004, he has been a Senior Leturer with the Shool of Information Tehnology and Computer Siene, University of Wollongong. His urrent researh interests inlude automati annotation and intelligent retrieval and adaptation of multimedia ontent, image/video analysis, multimedia seurity, multimodal biometris, and omputer vision. Chaminda Weerasinghe reeived BE Honors Class 1 with university medal from University of Wollongong, Australia in 1994 and his Ph.D. in image proessing from University of Sydney, Australia in He is a reipient of many aademi awards and medals from IEE, IEAust and IESA. Dr. Weerasinghe was with Motorola Australian Researh Center ( ), as a senior researh engineer. He is urrently with Toshiba (Australia) Pty. Ltd. (R&D Division). His main researh interests are in olor image proessing; CMOS image sensors, surveillane amera systems, stereosopi/panorami video generation/display and raster image proessing for printers.