A Prototype of Autonomous Intelligent Surveillance Cameras

Transcription

1 Universi of Wollongong Research Online Facul of Informaics - Papers (Archive) Facul of Engineering and Informaion Sciences 2006 A Proope of Auonomous Inelligen Surveillance Cameras Wanqing Li Universi of Wollongong, wanqing@uow.edu.au Igor Kharionenko Universi of Wollongong, igor@uow.edu.au S. Lichman Naional ICT Ausralia C. Weerasinghe Toshiba, Ausralia Publicaion Deails This aricle was originall published as: Li, W, Kharionenko, I, Lichman, S & Weerasinghe, C, A Proope of Auonomous Inelligen Surveillance Cameras, IEEE Inernaional Conference on Video and Signal ased Surveillance (AVSS '06), Sdne, November 2006, 101. Coprigh IEEE Research Online is he open access insiuional reposior for he Universi of Wollongong. For furher informaion conac he UOW Librar: research-pubs@uow.edu.au

2 A Proope of Auonomous Inelligen Surveillance Cameras Absrac This paper presens an archiecure and an FPGAbased proope of an auonomous inelligen video surveillance camera. The camera akes he advanage of high resoluion of CMOS image sensors and enables insanl auomaic pan, il and zoom adjusmen based upon moion acivi. I performs auomaed scene analsis and provides immediae response o suspicious evens b opimizing camera capuring parameers. The video oupu of he camera can be opimized o an region of ineres while he camera coninues o monior he enire scene. Field rials of he prooped camera have verified he proposed archiecure. Disciplines Phsical Sciences and Mahemaics Publicaion Deails This aricle was originall published as: Li, W, Kharionenko, I, Lichman, S & Weerasinghe, C, A Proope of Auonomous Inelligen Surveillance Cameras, IEEE Inernaional Conference on Video and Signal ased Surveillance (AVSS '06), Sdne, November 2006, 101. Coprigh IEEE This conference paper is available a Research Online: hp://ro.uow.edu.au/infopapers/504

3 A Proope of Auonomous Inelligen Surveillance Cameras Wanqing Li, Igor Kharionenko Serge Lichman Chaminda Weerasinghe Universi of Wollongong Naional ICT Ausralia Toshiba Ausralia P Ld {wanqing, igor}@uow.edu.au serge.lichman@nica.com.au cweerasinghe@oshiba.com Absrac This paper presens an archiecure and an FPGAbased proope of an auonomous inelligen video surveillance camera. The camera akes he advanage of high resoluion of CMOS image sensors and enables insanl auomaic pan, il and zoom adjusmen based upon moion acivi. I performs auomaed scene analsis and provides immediae response o suspicious evens b opimizing camera capuring parameers. The video oupu of he camera can be opimized o an region of ineres while he camera coninues o monior he enire scene. Field rials of he prooped camera have verified he proposed archiecure. 1. Inroducion Video surveillance and monioring ssems have become imporan componens in he modern securi infrasrucure. More and more cameras are insalled o provide efficien surveillance, bu his also requires emploing a sufficien number of skilled personnel for monioring. According o [1], he mos vulnerable par of video surveillance ssems is video monioring personnel. Mos of he ime, here are no alarming evens and he saff ma graduall loose concenraion on du. When alarming evens are capured b he ssem, he human response is ver ofen delaed and is no opimal. Anoher problem ha reduces efficienc of video surveillance ssems is a conradicion beween he required area of view and he sharpness of he capured objecs. The wider area capured b a camera, he less resoluion i can provide o represen deailed objecs. As a resul, image quali is usuall no sufficien o recognize facial or oher imporan feaures. One ma conclude ha he fundamenal problem resricing broader uilizaion of video surveillance ssems is caused b he concep when he ssem is considered onl as an observaion and recording device, enirel relied on human aenion and decision-making. There is a disinc rend in video surveillance marke owards using inelligen ssems, which are epeced o provide efficien assisance o he operaors. The core of hese ssems is auomaic scene analsis, which can be efficienl implemened using disribued cooperaive archiecures wih appropriae level of inelligence a differen levels. In his regard, developmen of echnologies and elecronic componens for inelligen video surveillance cameras ha can auomaicall opimize heir parameers and erac criical informaion for furher processing becomes an imporan pracical issue. This paper presens an archiecure and an FPGAbased proope of auonomous inelligen video cameras ha can urn passive surveillance ssems ino acive collaboraors o suppor securi operaors for immediae and efficien response o suspicious evens. 2. Camera Archiecure Figure 2 shows he archiecure of proposed inelligen surveillance camera. The camera incorporaes an auomaic pan, il and zoom (PTZ) funcionali implemened based upon [2,14] wih he parameers decided upon capured acivi wihin a specified region of ineres (ROI). This provides more deailed visual informaion in zoomed modes while monioring he full view a he same ime. The auomaic PTZ adjusmens can be insanl made based upon moion acivi, color and illuminaion changes of argeed objecs. The camera is buil upon a 1.3-megapiel SXGA CMOS image sensor wih digial oupu. Such high image resoluion allows displaing Zoom-1 (normal zoom, whole image), Zoom-2 and Zoom-4 wihou compromising resoluion a he video oupu. This is achieved elecronicall b cropping a required area of he image sensor jus wihin 40ms. The oupu video frame from he Color Processing module is per PAL field ( inerleaved), or digial YUV forma for H.263 video codec, depending on he surveillance ssem archiecure. The ROI Module alwas uses ROI daa for monioring of suspicious evens regardless of he currenl displaed area. Proceedings of he IEEE Inernaional Conference

4 Moreover, PTZ values are direcl used o opimize Color Processing Module parameers in such a wa ha no specific scaling sep is performed before or afer he inerpolaion or color reconsrucion of he video frame. Adapable image enhancemen and noise reducion feaures are implemened on he inerpolaed YUV color space wihin he Image Enhancemen module. These funcions applied onl o he luminance channel (Y), resul in significan improvemen in image quali, especiall while he camera operaes in poor lighing condiions. 3. PTZ ased Color Processing As well known, CMOS image sensors suffer from high level noise and piel cross-alk [2,14]. In order o achieve high quali images ha are comparable o CCD image sensors, a new color processing chain, as shown in Figure 3, was proposed. R Gr R Gr R R Gr R Gr R R Gr R Gb Gb Gb Gb Gb Gb Figure 1. Color filer arra (CFA) wih aer paern In he pre-processing module, he green channel is spli ino wo planes, which separaes he Gr (as shown in Figure 1) and he Gb values. All subsequen processing is carried ou separael for he red channel aer paern Preprocessing Module R Gb Gr and he blue channel using he associaed green Gr Color Correcion Module R Rc c Gbc Grc Color Inerpolaion Module Figure 3. Proposed Color Processing Chain RG The color correcion is performed on he aer paern using a neighborhood-based algorihm. The inerpolaion algorihm akes ino accoun he correlaion among he differen color channels and he PTZ values calculaed from he currenl ROI ha is auomaicall defined b he moion deecion and racking. The advanage of his color processing chain is ha i provides a beer quali oupu wih minimum noise escalaion using he proposed archiecure. However, he spliing of he green channel ino separae Gr and Gb channels creae non-sandard inerfaces beween he various modules. 3.1 Color Correcion Color correcion is performed using an algorihm developed b he auhors, which is described in deail in [6]. The Green channel is unalered. Red and lue channels are correced using he following formulae. = R R G (31) R c ( ) ( G ) c = (32) G c = G (33) The R, G, values represen he average channel value compued for each color filer arra (CFA) window of size 4 4. The resuling values are clipped o be wihin he range [0 255]. 3.1 Color Inerpolaion Color inerpolaion is performed on he aer paern aking ino accoun he pan il and zoom saus of he curren ROI. This approach can be used boh for up and down sampling wihou performing a specific scaling operaion. Man mehods are described for down scaling in a CFA paern b sub-sampling [3]- [5]. However, no mehods are described for boh up and down sampling of inerpolaed ri-color daa aking Sensor Clock Domain SDRAM Video Clock Domain Image Sensor (CMOS, SXGA) Sensor Daa & Conrol Inerface Whie alance Daa uffering Color Processing Image Enhancemen Video sream Video Encorder ROI (Moion Deecion & Tracking) Eposure Conrol Ssem Saus Regisers Communicaion s (Serial / IP based) Commands & Saus Figure 2. The archiecure of he proposed inelligen video surveillance camera channel values from he Gr plane and Gb plane respecivel. ino consideraion he relaive locaions of R, G and componens in he capured CFA. This implemenaion uses a mehod of weighed bilinear inerpolaion Proceedings of he IEEE Inernaional Conference

5 scheme based on relaive disance informaion from he inended inerpolaed piel locaion o he original R, G and componens in he CFA. The weighs are based on he zoom mode and he disance from he inended inerpolaed piel locaion o he original R, G and componens in he CFA. The sar addresses for accessing he CFA daa are deermined b he curren pan and il values; whereas he address incremen rae is deermined b he curren zoom value. Zoom values 1,2 and 4 are implemened in he camera [2,14]. The mehod of address incremen ogeher wih adapive weigh assignmen avoids he usual scale facors associaed wih convenional scaling. Such scale facors are picall floaing poin numbers and also involve division operaions, which are nonamicable for hardware implemenaion. The pan (P) and il (T) values are also consrained b he curren zoom (Z) value as shown in he following equaions (1) and (2). 1 0 P < (1) Z 1 0 T < (2) Z The processing CFA daa window is alwas se o be 4 4 regardless of he zoom value. This is also a hardware friendl feaure, since onl he mos demanding process deermines he hardware resources. In he Zoom-1 mode, CFA frame of size is inerpolaed o produce a full color frame of size In he Zoom-2 mode, CFA frame of size is inerpolaed o produce a full color frame of size In he Zoom-4 mode, CFA frame of size is inerpolaed o produce a full color frame of size Image Enhancemen 4.1 Sharpness Enhancemen Mehods of sharpness enhancemen found in lieraure [7][8] are in he cone of compression algorihms (e.g. JPEG), whereas a few deal direcl on he luminance (Y) and chrominance (U/V) signals [9]. Under hardware consrains, a simple non-linear and hardware-friendl sharpening filer ha proved is efficienc in [2,14] was implemened and emploed in YUV color space. The filer has he following properies: Filer aps: S 2, S 1, S 0, S + 1, S + 2 Y = S1 S 1 Filer coefficiens: If ( Y > 20 ) hen {-0.25, -0.25, 2.00, -0.25, -0.25} else {0.00, 0.00, 1.00, 0.00, 0.00}. Alhough he sharpening is performed onl on he Y channel, i is imporan o sore he corresponding U and V values o avoid color arifacs. This mehod performs well for naural images o enhance he appearance of sharpness o he human viewer wihou significanl increasing he noise level. 4.2 Noise Reducion The implemened noise suppression algorihm is based on anisoropic diffusion [9,10,11]. Anisoropic diffusion is performed along each video line, for Y channel onl. The main reason for his is o minimize he daa buffers needed for soring inermediae daa beween ieraions. Onl 3 ieraions are performed using inermediae sorage. Alhough onl Y channel is processed, corresponding U and V values should also be saved, o avoid color shifs a he oupu image. A line based anisoropic diffusion algorihm for noise reducion while preserving he edge sharpness was developed and implemened [2,14]. The correcion weighs for each piel Y value are based on is immediae horizonal gradien. Consider he following scenario: Y 2, Y, Y TheY which replaces he Y is compued as follows: = Y Y (3) 1 2 = Y Y (4) Y = Y (5) C 1 C 2 5. Moion Driven PTZ Moion analsis and objec racking have been sudied for several decades [12,13]. Due o he hardware consrains, a simple and effecive algorihm was proposed and implemened for deecing and racking he moion area o conrol he PTZ parameers of he camera. Figure 4 shows is flow char. Proceedings of he IEEE Inernaional Conference

6 5.1 Frame Size Reducion To reduce he frame capured wih piel resoluion, i is subdivided ino a number of blocks 6464 piels each. Therefore here are 320 (20 16) blocks (m, calculaed according o (6). k 1 k 1 1 ( m, = f ( km+, kn+ ) (6) 2 k = 0 = 0 where f(, ) is he iniial high-resoluion frame acquired from image sensor and k is he block size, which is equal o 64. In oher words, he reduced size frame is obained b block based averaging of he capured high-resoluion frame. Onl he reduced size frame (2016) is used hen for moion acivi deecion. 5.1 Frame Difference Calculaion D ( m, 3D ( m, + ( ( m, ( m, ) = (7) Calculaion of ROI If he absolue value of D (m, is equal or greaer han he hreshold TH, hen he corresponding 6464 block is indicaed as a moion acivi area. Is alarm flag A (m, is calculaed according o (8). A 1 D m n TH = (, ) ( m, (8) 0 oherwise ROI is defined as he one ha includes all blocks, which have non-zero A (m, according o (9). l m r m n b n = min( m A ( m,*) > 0) = ma( m A ( m,*) > 0) = min( n A (*, > 0) = ma( n A (*, > 0) (9) where superscrips l, r,, b relae o he lef, righ, op and boom block boundaries of he ROI respecivel. The zoom facor Z can be calculaed using he following (10). Z = 1, r m l W m > 2 r l W 4, m m < 4 2 oherwise OR b n H n > 2 b n H n < 4 (10) where W is he widh and H is he heigh of blocks wihin he reduced size frame. W=20 and H=16 in our implemenaion. The posiion for panning P and iling T are defined in accordance o (11) and (12) Figure 4. A flow char of he moion deecion and objec racking algorihm The frame difference calculaion is based on he weighed difference algorihm, which akes ino accoun he block values from several consecuive frames. Such emporal filering reduces he influence of luminance changes due o he noise. The difference frame has size 2016 and consiss of he values D (m, calculaed according o (7). c W P = m (11) 2 * Z c H T = n (12) 2 * Z c l r where m = ( m + m ) / 2 is he cenral posiion of c b he ROI in horizonal direcion and n = ( n + n ) / 2 is he corresponding cener in verical direcion. Proceedings of he IEEE Inernaional Conference

7 5.3 allisic Smoohing Eperimens wih he inelligen camera [2,14] showed ha simple applicaion of he panning and iling values, P and T, calculaed from equaions (11) and (12) for reposiioning ROI had resuled in a jiered video sequence, especiall when several parameers have o be changed a same ime. In order o sabilise he sequence, he concep of ballisic smoohing was inroduced, which implemens a second-order low-pass emporal filer. The smoohing is applied o he cenre posiion of he deeced moving ROI a he original sensor resoluion. Firs, he ROI cenre posiion is convered from block window space o piel space using equaions (13) and (14). = m * k ( k / 2) (13) + n k + = * ( k / 2) (14) Then, he acceleraion of he moving ROI (in piels per frame) is calculaed as follows. = ± s (18) The coordinaes of he new ROI ogeher wih he zoom facor are sen o he Colour Processing Module ha selecs he corresponding area and applies PTZ based color inerpolaion. 6. Proope and Resuls Figure 5 shows a proope of he inelligen camera. I was esed in differen surveillance scenarios b securi professionals. These scenarios include boh indoor and oudoor scenes, such as monioring of he airpor air raffic from he disance, monioring of he vehicles a he highwa, surveillance a he ehibiion pavilion, hallwa and lab monioring. Tes condiions, es equipmen and he obained resuls are described in deails in [2,14]. The ess showed ha camera resoluion was up o 800 lines. I had a quick response o he suspicious acivi, a a 2, 1 > Disma = 1, > Dis min 0, oherwise 2, > Dis ma = 1, 1 > Dis min 0, oherwise s 1 < Sp MAX c c < Dis ma s < Sp ma 1 < Disma s < Sp ma s 1 < Sp ma (49) (50) Figure 5. A proope of he auonomous inelligen camera where, are coordinaes of he ROI in he previous frame, s, s are he speed of he ROI 1 1 calculaed in he previous frame, Sp ma is he maimum permied speed of he ROI and Dis min, Dis ma are minimum and maimum allowable ravelling disances of he ROI beween previous and curren frames. The speed of he ROI (in piels per frame) is defined as s = s ± a (15) s = s ± a (16) The posiion of he new ROI is calculaed using equaions (17) and (18). = ± s (17) auomaicall zooming ino he opimal size ROI, which encloses all moving objecs and racking hese objecs when he move. I should be poined ou ha he camera coninues monioring he enire scene while displaing he ROI. If moion acivi is deeced beond he curren ROI, he camera can auomaicall adjus he area size. Thus, he objecs are capured b he camera wih he maimum possible resoluion ha faciliaes furher auomaic high-level analsis, such as human acivi idenificaion [15][16]. Figure 6 shows he hallwa racking sequence when one of he auhors walked from he end of hallwa owards he camera and hen disappeared. The whie bars on he righ boom side of he frames show he zoom facor (one bar for zoom 1, or he whole view field, wo bars zoom 2 and four bars for zoom 4). The blinking red bar in he op righ corner of he frames indicaes ha he moion was deeced. 7. Conclusion The described archiecure was implemened on a one million gaes FPGA-based plaform. Complei of he FPGA-based implemenaion indicaed Proceedings of he IEEE Inernaional Conference

8 feasibili of inegraion of he described soluion o an inelligen surveillance ssems. 10. References [1] C. Regazzoni, G. Fabri, G. Vernazza. Advanced Video- ase Surveillance Ssems. Kluvwer Academic Publisher [2] C. Weerasinghe, W. Li, M. Nilsson I. Kharionenko. Digial Zoom Camera wih Image Sharpening and Noise Reducion. IEEE Transacions on Consumer Elecronics, Vol. 50, pp , Augus 2004 [3] M. Guarnera e al, Mehod for processing digial CFA images, paricularl for moion and sill imaging, U.S. Paen Applicaion No. 2003/ A1 [4] H. Fukuda, Image pickup apparaus, U.S. Paen Applicaion No. 2003/ A1 [5] K.A. Parulski, Color filers and processing alernaives for one-chip cameras, IEEE Trans. on Elecron Devices, Vol. ED-32(8), pp. 1381, 1985 [6] I. Kharionenko, S. Twelves and C. Weerasinghe, Suppression of noise amplificaion during color correcion, IEEE Trans. Consumer Elecronics, Vol. 48 (2), pp , 2002 [7] M.Fischer, J.L. Paredes, G.R. Arce, Weighed median image sharpeners for he World Wide Web, IEEE Trans. on Image Processing, Vol. 11 (7), pp , 2002 [8] K. Konsaninides, V. haskaran, G. erea, Image sharpening in he JPEG domain, IEEE Trans. on Image Processing, Vol. 8 (6), pp , 1999 [9] S. J. Huang, Adapive noise reducion and image sharpening for digial video compression, Proc. of IEEE Inernaional Conference on Ssems, Man, and Cberneics, Vol. 4, pp , Oc., [10] Perona P. and Malik J., Scale-space and edge deecion using anisoropic diffusion, IEEE Trans. Paern Anal. Mach. Inell., Vol. 12, No. 7, PP , Jul [11] lack M.J. and Marimon D.H., Robus Anisoropic Diffusion, IEEE Trans. On Image Processing, Vol. 7, No. 3, pp , March [12] D.J. Connor and J.O. Limb. Properies of framedifference signals generaed b moving images. IEEE Trans. Communicaions, COM-22(10): , [13] Y.W. Huang,.Y. Hsieh, S.Y. Chien, and L.G. Chen, Simple and effecive algorihm for auomaic racking of a single objec using a pan-il-zoom camera. Proc. of ICME 2002, Swizerland. [14] Weerasinghe, C.; Wanqing Li; Kharionenko, I.; Nilsson, M.; Twelves, S. Novel color processing archiecure for digial cameras wih CMOS image sensors, IEEE Trans. Consumer Elecronics, 51(4), pp , [15] Wei Niu; Jiao Long; Dan Han; Yuan-Fang Wang, Human acivi deecion and recogniion for video surveillance, 2004 IEEE Inernaional Conference on Mulimedia and Epo, Vol. 1, pp [16] N.D. ird, O. Masoud, N. P. Papanikolopoulos, Deecion of loiering individuals in public ransporaion areas, IEEE Transacions on Inelligen Transporaion Ssems, Volume 6, Issue 2, pp , June (1) zoom 1 (2) zoom 1 (3) zoom 4 (4) zoom 4 (5) zoom 2 (6) zoom 2 (7) zoom 2 (8) zoom 2 (9) zoom 1 (10) zoom 1 Figure 6. Hallwa moion racking scenario Proceedings of he IEEE Inernaional Conference