Event Detection Using Local Binary Pattern Based Dynamic Textures

Transcription

1 Event Detetion Using Loal Binary Pattern Based Dynami Textures Abstrat Deteting susiious events from video surveillane ameras has been an imortant task reently. Many trajetory based desritors were develoed, suh as to detet eole running or moving in oosite diretion. However, these trajetory based desritors are not working well in the rowd environments like airorts, rail stations, beause those desritors assume erfet motion/objet segmentation. In this aer, we resent an event detetion method using dynami texture desritor. The dynami texture desritor is an extension of the loal binary atterns. The image sequenes are divided into regions. A flow is formed based on the similarity of the dynami texture desritors on the regions. We used real dataset for exeriments. The results are romising. 1. Introdution Yunqian Ma Honeywell International In Douglas Drive North Golden Valley, MN 55442, USA yunqian.ma@honeywell.om The event reognition in video surveillane is an imortant researh toi. Normally the objet (suh as a erson or a ar) is deteted, and traked as a single objet. Then a simle ativity reognition, inluding eole walking, running, skiing, an be erformed [1][2]. However, the assumtion that objets an be searated and traked often fails in the rowd environment. For examle, surveillane ameras in airort an ature a lot of ersons walking together. Thus the ommonly seen ativities, suh as eole moving in different diretions, rowd formation and disersal, beome hard to be deteted and be reresented reliably. There is muh interest to the event detetion in rowd environment. Ali and Shah [3] develoed an ativity reresentation by modeling the rowded sene as a fluid flow to byass objet detetion and traking. They used the Lagrangian Partile Dynamis to segment high density rowd flows and detet flow instabilities. Andrade et al. [4] roosed a tehnique to automatially detet abnormal events in rowds. They haraterized rowd behavior by observing the rowd otial flow and used unsuervised feature extration to enode normal rowd behavior. Marana et al. [5] resented a tehnique for rowd density Petr Cisar Honeywell Prague Labs V Parku 2326/18, 14800, Prague, Czeh Reubli etr.isar@honeywell.om estimation based on Minkowski fratal dimension. Reisman et al. [6] resented a real time system that was able to detet rowds at distanes of u to 70m. The system used slies in the satiotemoral domain to detet inward motion as well as intersetions between multile moving objets. Ma and Cisar used dynami texture for eole segmentation in the rowd environment [7]. For a more omrehensive survey on ativity reognition in rowded environments, we refer the reader to [8]. In this aer, we resent an event detetion method using the dynami texture desritor. The dynami textures are the textures with motion. Seifially, we use the Loal Binary Patterns from Three Orthogonal Planes (LBP-TOP) desritor. We first artition the image frame into multile regions. Eah region and the onseutive several frames regions form a volume. The LBP-TOP desritor is alulated on the volume. Next, we form a flow by onneting several volume regions together along the time axis. Then the desritor for the event is reresented by the LBP-TOP desritors along the flow. For the lassifiation on the event, the distane between the test sequene s desritor and the model sequene s desritor is alulated using the log-likelihood statistis. This allows us to reognize events of eole and ars in different video senarios. This aer is organized as follows. Setion 2 gives an overview of dynami texture desritor. Setion 3 resents the roosed methods using the dynami texture for the event reognition. We resent exerimental results in Setion 4. Setion 5 is disussion. 2. Dynami texture desritor 2.1. Previous work In this setion, we briefly desribe the revious work for the Loal Binary Pattern (LBP), its extension to the LBP-TOP desritor [9][10][11][13][14][15]. The LBP was introdued as a texture desritor [9][10]. An LBP desritor for a loal neighborhood on a enter ixel is alulated with the eight neighbors using the grey level of the enter ixel as a threshold. The LBP desritor atures the satial struture of loal image texture, and has been suessfully used in the aliation of texture lassifiation [10] /09/$ IEEE 38

2 Zhao and Pietikäinen [11] extended the LBP desritor to Volume LBP to ature the dynami texture on the image sequenes. However, the number of atterns will beome very large when the number of neighboring ixels inreases. Therefore the LBP-TOP desritor was develoed [13] for a fast alulation of the LBP features for dynami texture. The LBP-TOP desritor is a onatenation LBP on three orthogonal lanes, XY, XT and YT. Zhao and Pietikäinen [14] alied the LBP-TOP desritor for the aliation of faial exression reognition. They first divided the fae image into several regions. The LBP-TOP desritor is alulated for eah region. They viewed the faial exression reognition roblem as a lassifiation roblem. The distane between a samle histogram and the model histogram is alulated using the log-likelihood statistis. Kellokumu, Zhao and Pietikäinen [15] alied the LBP-TOP desritor for ation reognition inluding eole bending, juming when a erson is well seerated. They only used the XT lane and the YT lane to alulate the LBP-TOP desritor LBP-TOP desritor Before we resent the LBP-TOP desritor, we first resent the LBP desritor. The original LBP desritor on a enter ixel is alulated with the eight neighbors using the grey level of the enter ixel as a threshold. Ojala et al. [10] defined a texture T in a loal neighborhood of a gray image as a joint distribution of the gray level of the enter ixel and those of its neighborhood ixels T = t( g, go,..., gp 1) (1) where g is the gray level of the enter ixel and P is the number of neighborhoods. Then we subtrat the gray level of the enter ixels and fatorize the joint distribution. T t( g) t( go g,..., gp 1 g) (2) Normally t( g ) is omitted, sine it reresents the whole luminane of an image. Moreover, in order to have the invariane with the sales of the gray level, only the signs of the differene are onsidered, so the LBP ode is reresented as follows: P 1 = 0 LBP( x, y ) = s( g g )2, (3) 1, x 0 ( x) = 0, x < 0 s (4) The reason to all the LBP ode is beause a binomial weight 2 is assigned to eah indiation funtion between the gray level of the enter ixel with the gray level of eah neighborhood ixel. The texture in one image an be extended to dynami texture in one image sequene. Consider a volume neighborhood entered at ixel ( x, y ) at time t, the volume LBP is defined as the joint distribution of the gray levels of 3P+3 image ixels on the urrent frame t, the revious neighboring frame, and the next neighboring frame t L t + L as follows: 3P + 1 = 0 V LBP( x, y, t ) = s( g g )2 (5) where g reresents the gray level of the enter ( x, y ) at time t, and g reresents the gray level of the neighborhood ixels within the satiotemoral volume. (a) (b) Figure 1: Illustration of the LBP-TOP desritor (a) Calulation of three LBP odes for eah oint of dynami texture through sanning, (b) histograms of LBP odes of three orthogonal lanes. When the number of neighborhood inreases, the 3 2 number of atterns for the VLBP inreases by 2 P +, so the number will be very large, next we resent the LBP- TOP desritor [13]. The LBP-TOP desritor is a desritor onatenating the LBP on the three orthogonal lanes: XY lane, XT lane, and the YT lane. In Figure 39

3 1(a), the XY lane reresents the image frame; the XT lane reresents the dynamis in the horizontal diretion, and the YT lane reresents the dynamis in the vertial diretion. In Figure 1(b), the LBP ode is alulated on the XY, XT, and the YT lanes. The XY-LBP ode reresents the satial information, and the XT-LBP ode and the YT- LBP ode reresent the satial temoral information. The 3 2 number of bins is only 3 2 rather than 2 P+ for VLBP. In LBP-TOP, all the ixels are alulated as the enter ixel, and then the statistis is reresented as a histogram on eah lane. The three histograms are onatenated into one histogram as LBP-TOP desritor. H = I{ LBP( x, y, t) = i (6) } i, j x, y, t 1 I ( A) = 0 ( x, y, t A_ is _ true o. w. where LBP ) reresents the LBP ode of enter ixel (x, y, t) in the jth lane (j = 0: XY lane, j = 1:XT lane, and j = 2: YT lane). The radius in the time axis does not need to be the same as the radius in the sae axis. We denote the radius in axe X is R, the radius in axe Y is R, and the radius X in axe T is R T. Also, the number neighborhood oints in XY, XT and YT lane orresonds to P XY, P XT, and P YT. Then the notation of the LBP-TOP desritor an be reresented as LBP TOP P XY, PXT, PYT, RX, RY, R. T The distane between two LBP-TOP desritors an be measured using the log-likelihood statisti as follows: B b d( H, H ) H b log H (7) = = where B is the number of bins, Y 2 b H 1 b reresents the samle s robability at bin b for Histogram 1, and 2 H b reresents the samle s robability at bin b for Histogram 2. Other distane between the two histograms, suh as histogram intersetion distane an be used. 3. Proosed method In this setion, we resent a region based method using the LBP-TOP desritor. We artition the image frame into regions. Eah region in the urrent frame and its next several frames form a satio-temoral volume, as shown in Figure 2. Eah satio-temoral region is a three dimension of width X, height Y, and number of time frames T. The satio-temoral regions an be overlaing or non-overlaing. If two satio-temoral regions overla, we denote Xo, Yo as the distane between enters of the regions in X axe and Y axe resetively, and To as the differene between entral frames in T axe. Figure 2: Image regions from onseutive frames. The LBP-TOP desritor of the region is alulated using the method (6) in Setion 2. The event may over many frames, whih is longer than the number of frames forming the volume region, so the regions are onneted temorally to form a flow. There are three omonents to form a flow using the regions: temoral assoiation of the athes, a sto riterion for the temoral assoiation, and flow editing to remove noise. Temoral assoiation is to onnet a region at time t to a region at time t+1. We assume that the ath at time t + 1 that is onneted to a ath B i (t) at time t lies in its satial neighborhood. A ath Bˆ j ( t + 1) is assoiated with B i (t) ˆ j = argmin d(h( B i (t)), H( Bj (t + 1))) (8) where the d(.) is the distane between the two LBP-TOP desritors (7). This roedure is ontinued until a stoing riterion is satisfied. We onsidered two stoing riteria. First, if all nine neighboring athes at time t + 1 do not exhibit any motion, we assume that the objet either stos moving or has left the sene. The orresonding flow forming is eased. Seond, if objets overla leading to the olusion of the objet in onsideration, the LBP-TOP desritors of all nine neighborhood athes will differ greatly with the LBP-TOP desritors of the urrent ath B i (t). In artiular, if the distane >μ+3δ, where μ is the mean and δ is the standard deviation of the distanes alulated during the flow formation, the orresonding flow is eased. Figure 3 shows an examle of flows formed in an olude situation using the seond sto riteria. The uer three image frames in Figure 3 reresents an image sequene. The first image frame shows two ersons move towards in the left of the image, the seond image frame shows the two ersons get oluded, and the third image frame shows the two ersons then searate. The lower 40

4 image in Figure 3 resents the flows orresonding to the oluded erson get divided at the time of the olusion using the stoing riteria desribed above. So there are two red flows orresond to the erson got oluded. Figure 3: Uer images are an examle of overlaing movement; lower image is the formed streamlines, the overlaed objet is divided into two arts aording to the sto riteria After that, we erform flow editing. We filter out three tyes of noise. First, we filter out the flows with the starting region on the boundary of the image. Those flows may join neighboring regions out of the boundaries of the image in the next few frames. Seond, flows with no signifiant hange of oordinates are onsidered noisy flows and are removed. These flows orresond to isolated movement observed in the bakground. Third, surious flows with their lengths shorter than a redefined threshold are filtered out. The resulting flows are a robust set of flows that an be used for the event detetion. Eah flow is reresented by the LBP-TOP desritors along the flow. The distane between the two flows is defined as a distane for lassifiation of the flows event. Suose a test flow Sq and a model flow S in the database, we an alulate n 1 d ( Sq, S ) = d( H( Bq ( t)),h( B ( t))) (9) n t= 1 where n is number of the LBP-TOP desritors in eah flow. The LBP-TOP desritors for eah volume region along the flow desribe the dynami information of the flow. 4. Exerimental results In this setion, we resent the exerimental results for the roosed event reognition method. We onduted the exeriments on three sets of data. The first one is the edestrian dataset used in [16]. The seond one is the retail esalator data and the vehile traffi data from the UCF dataset [3]. The third data is a subway dataset from ilid dataset [17]. The edestrian dataset [16] was used for the first exeriment. The dataset ontains video of high traffi of edestrians from a stationary amera. The resolution of the video sequene is 238*157 at 10 frames er seond. The edestrians in the dataset are moving in two diretions: moving right-u and moving left-down. We extrated the art of the data where high density of the rowd is resented with many olusions of the edestrian. We use 22 flows eah with 7 frames for training of the LBP-TOP desritor for two ativities. For eah tye of event, 11 flows ontaining the ativity were seleted. The model was alulated and stored in the database. We used 400 image frames for testing the roosed method. The video sequene was divided into satio-temoral overlaing regions with the size X = 8, Y = 9 ixels and T = 7 frames of overlaing Xo = 4, Yo = 4, and To = 1. The arameter setting for the LBP-TOP desritor is LBP- TOP 10,10,10,5,5,3. In the testing hase, eah region was lassified into one of the trained lasses using the distane defined by (9). Figure 4 shows the event reognition results. The red athes show the eole moving right-u diretion, while the green athes show the eole moving left-down diretion using our method. The results showed that the method we resented is suitable for the edestrian dataset. In the seond exeriment, we used the UCF dataset [3]. We use two sets of data from this dataset: the retail esalator data and the vehile traffi data. Figure 5 shows two frames of the event reognition results on the retail esalator set of videos. Figure 6 resents the event reognition results on the vehile traffi set of videos from the dataset. Eah region is olored by the olor that orresonds to the olor of the event lass. In the third exeriment, we used the i-lid dataset. The i-lid dataset onsists of video data of subway latform where eole enter or leave the train. Different levels of eole density are resented in the video data, and eole are overla often. The original resolution of the video data is 720*576 with 25 frames er seond. The video data was down samled the video data to 188*144 with 12.5 frame er seond for faster roessing. 41

5 X = 15, Y = 13 ixels and T = 5 frames with overlaing Xo = 6, Yo = 6 and To = 1 and desritor LBP- TOP 9,9,9,5,5,3. The otimal width and height of satiotemoral volume should orresond to the ¼ of the objets in the sene and the overlaing should orresond to the ½ of the size of the volume. The lassifiation auray is very good for these four ativities. The events are not simle beause it ontains many overlas of the objets. TABLE I CLASSIFICATION ACCURACY - SUBWAY DATASET Ativity Down U Left Right Down 96% 0% 4% 0% U 0% 96% 0% 4% Left 30% 0% 70% 0% Right 0% 0% 0% 100% Figure 4: Samle results of ativity reognition into two lasses using the LBP desrition, red = athes with right-u motion, green = athes with left-bottom motion. We seleted n video sequenes for eah different ativity (lass). Eah sequene was divided into satiotemoral regions, and the flows were formed. The flows were formed for eah testing sequene and eah flow s event was automatially lassified by roosed aroah. After that we omared with the ground truth to alulate the onfusion matrix as shown in the Table I. There are four events in the dataset: eole moving u, eole moving down, eole moving left, and eole moving right. Figure 7 shows examles of the event in the subway dataset for the orresonding event. For eah lass we had more than 16 flows for training. For testing we have 4 sequenes and 83 flows on the average for eah lass. In the table I we resent the onfusion matrix for the four tyes of events. These results were obtained with following settings: satio-temoral volumes with the size 5. Disussion Event detetion in video surveillane is imortant. The trajetory based desritor, suh as satial oordinate, veloity, and shae works if the individual objet an be segmented and traked. However, the high density environment is ommon in video surveillane data, where the trajetory based ativity desritor work oorly in this environment. In this aer, we resent an event detetion using the dynami texture desritor. The dynami texture atures the stationary roerties in time [18]. We use the LBP- TOP desritors. We first artition image sequenes into regions. Then we form motion flow by temorally onneting the region in the urrent frame to the ath in the next frame. Event reresentation is from the LBP-TOP desritors extrated from the flow. We use various real data sets to test the roosed method. The exerimental results show good erformane for the event reognition. Referenes [1] Y. Ma, S. B. Damelin, O. Masoud, N. Paanikolooulos, Ativity Reognition Via Classifiation Constrained Diffusion Ma. In: International Symosium on Visual Comuting,. 1 8, 2006 [2] Y. Ma, B. Miller, P. Buddharaju, M. Bazakos, Ativity Awareness: From Predefined Events to New Pattern Disovery, In: IEEE International Conferene on Vision Systems, New York, NY, USA, January 5-7, [3] S. Ali, M. Shah, A Lagrangian Partile Dynami Aroah for Crowd Flow Segmentation and Stability 42

6 Analysis, in IEEE Conferene on Comuter Vision and Pattern Reognition, Minneaolis, MN [4] E. Andrade, S. Blunsden, and R. Fisher, Detetion of emergeny events in rowded senes. In IEEE International Symosium on Imaging for Crime Detetion and Prevention, [5] A. Marana, L. Costa, R. Lotufo and S. Velastin, Estimating Crowd Density with Minkowski Fratal Dimension, IEEE International Conferene on Aoustis, Seeh, and Signal Proessing, [6] P. Reisman, O. Mano, S. Avidan, A. Shashua, M. Ltd, and I. Jerusalem, Crowd detetion in video sequenes, IEEE Intelligent Vehiles Symosium, [7] Y. Ma and P. Cisar, Motion Analysis Using Dynami Texture in Crowd Environment, Image Analysis - From Theory to Aliations, Researh Publishing, 2008, [8] B. Zhan, N. Monekosso, P. Remagnino, S. Velastin, and L. Xu, Crowd analysis: a survey, In Mahine Vision and Aliations, 2008, 19, [9] T. Ojala, M. Pietikäinen and D. Harwood, A omarative study of texture measures with lassifiation based on featured distributions, Pattern Reognition 29, , [10] T. Ojala, M. Pietikäinen, and T. Mäenää, Multiresolution gray-sale and rotation invariant texture lassifiation with loal binary atterns, IEEE Transation on Pattern Analysis and Mahine Intelligene, vol. 24, no. 7, , [11] G. Zhao and M. Pietikäinen, Dynami Texture Reognition Using Volume Loal Binary Patterns, Pro. Euroean Conferene on Comuter Vision, 2006 Worksho on Dynamial Vision, Graz, Austria. [12] htt:// [13] G. Zhao and M. Pietikäinen, Loal Binary Pattern Desritors for Dynami Texture Reognition, in International Conferene on Pattern Reognition, 2006, [14] G. Zhao and M. Pietikäinen, Dynami Texture reognition Using Loal Binary Patterns With an Aliation to Faial Exressions, in IEEE Transation on Pattern Analysis and Mahine Intelligene, vol. 29, no.6, , 2007 [15] V. Kellokumu, G. Zhao, M. Pietikäinen, Human Ativity Reognition Using a Dynami Texture Based Method, in British Mahine Vision Conferene (BMVC), 2008 [16] A. Chan and N. Vasonelos, Modeling, Clustering, and Segmenting Video with Mixtures of Dynami Textures, IEEE Transation on Pattern Analysis and Mahine Intelligene, vol. 30(5), , May [17] i-lids dataset for avss, [18] G. Doretto, A. Chiuso, S. Soatto, and Y. Wu, Dynami textures, International Journal of Comuter Vision, Vol. 51, no.2, , Figure 5: Ativity reognition shown for samle frames from the UCF retail esalator dataset. The red and green arrows denote the two major motion atterns observed in the video sequenes. The orresonding olor oded athes denote the labels oututted by our system. Figure 6: Ativity reognition shown for samle frames from the UCF traffi dataset. The red and green arrows denote the two major motion atterns. The orresonding olor oded athes denote the oututted ativity labels. The small resolution of objets in the sene auses some athes to be erroneously lassified. 43

7 Figure 7: Examles of four events in the i-lid dataset: erson moving down, erson moving left, erson moving right. 44