Complex Human Activity Searching in a Video Employing Negative Space Analysis

Transcription

1 Compex Human Activity Searching in a Video Empoying Negative Space Anaysis Shah Atiqur Rahman, Siu-Yeung Cho, M.K.H. Leung 3, Schoo of Computer Engineering, Nanyang Technoogica University, Singapore FICT, Universiti Tunu Abdu Rahman (Kampar), Maaysia shah008@ntu.edu.sg, davidcho@pmai.ntu.edu.sg, 3 asmeung@gmai.com Abstract - In this paper a region based technique is proposed to search compex human activity in a video. Most of the current activity recognition systems deas with singe action type but in this system, by taing meaningfu composition of different types of human actions, compex human activity is constructed which is searched in a video sequence. To extract features, this system empoyed the surrounding regions of the human sihouette which is termed as negative space according to art theory whereas other region based methods wor on the sihouette of the person. Negative space based features have the abiity to describe human poses with simpe shapes impying simpe description of poses. Moreover, negative space features are ess sensitive to noise, overcome some imitation of sihouette based methods such as eas or hoes in the sihouette, robust to partia occusion, shadows and different cothing. The system consists of hierarchica processing of bacground segmentation, region partitioning, extraction of shape based features, finding matching score by Dynamic Time Warping and speed cacuation. The system is compared with state of the art method and offered significanty improved performance over different compex activities. Keywords: Human action recognition; Negative space; Sihouette; Dynamic time warping; Compex activity. Introduction Automatic categorization of human activity is an important topic in computer vision. Soving of this probem is important due to its vast appication area which incudes virtua reaity, games, video indexing, human computer interaction, video surveiance etc. Most of the chaenges faced by human action recognition systems are cuttered bacground, camera motion, shadow, viewpoint change etc. Various wors have been proposed with diverse ideas to recognize human actions [, ]. Some of the methods have shown good performance but there are sti some imitations. Periodicity anaysis based methods [3] ony deas with cycic actions. Key frames or eigen-shapes of sihouettes methods [3, 4] do not utiize the motion information which is an important cue for some actions. Tracing based methods [5] face difficuties in case of sefoccusion, change of appearance etc. Bags of words methods [6] extract oca features which do not have the information of tempora co-reation between frames. On the other hand, region based methods wor we as they are robust to noise. Region based methods face difficuties in case of shadow and partia occusion []. Fig. : Pose description by means of negative space regions. (a) Input pose, (b) After region partitioning process, (c) Negative space regions represented as triange or quadrange. Most of the current systems of action recognition wor with the videos containing singe action type. However, in rea ife, human activity can be meaningfuy composed, e.g. waing foowed by running. In this paper, we proposed a region-based method which can search for compex human activity (composition of different human activity) in a singe video. To perform the matching, we extract features from the surrounding regions of human sihouette which is termed as negative space according to art theory [7]. In art theory, negative space is the space between the object (positive space) and the canvas. In our case, smaest bounding rectange of human sihouette is the canvas and the human sihouette is the positive space which indicates that the empty regions inside the bounding rectange are the negative space (Fig. (a)). By empoying negative space, poses can be described by simpe shapes which are formed by the sihouette and the bounding box as shown in Fig. (c). Moreover, negative spaces based methods are robust to noise since it s a region-based methods with other advantages, e.g. not affected by hoes or eas inside the sihouette, robust to partia occusion, shadows, cothing variations and deformed actions which is shown in our earier wor [7]. However, our earier wor [7] deas with singe action type. In this paper, we extend the idea of negative space to query for compex human activity in a singe video based on the simpe shape features and motion information extracted from the negative spaces.

2 Fig. : Boc diagram of the system. Proposed system Our proposed system s boc diagram is shown in Fig.. In our system, query activity can contain mutipe action type (e.g. wa, run) and the input video is a singe video sequence containing mutipe types of activities which are performed sequentiay. Our goa is to search the action types in the input video, in the same order as given in query activity (e.g. wa foowed by run). We are interested to find the action types ony in the input sequence regardess of their tempora position in the input sequence. Mode data and input data are processed separatey which share some common processing steps. Sihouettes of the human body are obtained by bacground subtraction in segmentation step. In our current system we empoyed Li et a. [8] agorithm to segment the bacground. Negative space regions are captured in negative space extraction step by taing the smaest rectange bounding the human sihouette of the segmented image. Inside the bounding box, empty regions (Fig. (a)) are the region of interest in our system. Compex negative space regions are partitioned into simpe regions in the region partitioning step. In pose description, shape based features are extracted from the negative space regions to describe poses. In speed cacuation step, speed of mode and speed of input subsequence is cacuated. In case of mode data, speed is cacuated from a the segmented images of a mode sequence whereas for input data, speed is cacuated from the segmented images of matched input sub-sequence ony which can be obtained from the matching step. Matching of the mode data with input video sequence is performed by Dynamic Time Warping (DTW) in matching step and tota matching score is obtained. 3 Region partitioning Number of regions shared by the same pose group may not aways be same because of continuous movement of the body as shown in Fig. 3 where pose 3(a) and 3(b) are taen from same pose group but they do not share same number of regions. Region partitioning is appied to overcome this situation and simpify the matching process. Technique used in [7] is empoyed for the partitioning process where by ine scanning process peninsua growing from the sihouette is identified. The vaidity of the peninsua for partition is identified by the protrusive measure of three distances (Fig. 3(c)). If the peninsua is vaid for partition, the region is partitioned into two by the tip of the peninsua (Fig. 3(d), 3(e)). Finay sma regions, yieded due to the different cothing or noise, are removed since those regions provide nonimportant description of the pose. 4 Pose description To describe each pose, we extract two types of features: positiona (ocate the empty region inside the bounding box) and region-based features (describe the shape of the region). For identifying the ocation of the region, we abe the bounding box with 4 anchoring points as done in our earier wor [7]. For each region, mid-point on the side of the bounding box is computed and the region is assigned a positiona abe with respect to the nearest anchoring point from that mid-point. Simpe region based features are extracted to describe the trianguar or quadranguar shape of the negative space as shown in Fig. (c). Our shape based features are area, orientation, eccentricity, rectanguarity, horizonta and vertica engths which are cacuated as foows:

3 Fig. 3: Region partitioning scenarios. (a) No partition is needed, (b) partition is desired (c) partitioning measures taen for region x of (b), (d) partition output of region x. (e) fina partion output of (b). Area: region area is cacuated by centraized moment, 00, which is normaized by tota empty area. Eccentricity: Simpe eccentricity measure is / where is the semi-minor axis ength and is the semi-major axis ength, which can be cacuated by foowing equations where ij is the i, j th order centraized moments of the region s pixes. Orientation: Ange of major principe axis (orientation of the shape), denoted by can be determined by µ, (µ 0 - µ 0 ) and [9], where is cacuated by tan 0 s range is -/ to / which is normaized to the range 0 to. Rectanguarity: Simpe rectanguarity measure is the ratio of region area over bounding rectange area of that region (equation (4)). rectanguarity=reg_area/bounding_rec_area (4) Horzonta and vertica side engths: The engths of the bounding box sides incuded in each region are used as features, which are normaized by height (ver en ) or width (hor en ) of the bounding box (equation (5) and (6)) 0 reg _ hor _ en hor en sign bb _ width reg _ ver _ en sign bb _ height ver en () () (3) (5) (6) where sign is negative for bottom and eft side of the bounding box, otherwise positive. For region with a corner point from the bounding box both hor en and ver en wi be present, or ese one of the engths wi be zero. Our feature vector ength is 6*4=84 as we have 4 anchoring points and for each region there are 6 features. 5 Speed cacuation Speeds of the person are important cues in activity recognition since different actions are performed in different speeds (e.g. running is performed in faster speed than waing). Speed of the person coud be cacuated as equation (7) hor sp ds dt xi xi xi xi fr _ rate (7) fr _ rate where x i is the X-axis coordinate of the centroid of human body of i th frame and fr_rate is the frame rate of the sequence. Equation (7) is normaized by the height of the bounding box of the person to remove scaing effect. Expressing with respect to the average vaue we have t _ hordisp fr _ rate n t hor fr rate hor _ disp _ sp t _ height t _ height n where t _ hor disp n i x x i i (8), n is the tota number of frames in the sequence and t _ height is the sum of a bounding box height excuding the first frame. For mode data a the segmented images are taen to cacuate the speed whereas for input data ony the matched poses are taen which are obtained by matching with certain mode sequence. 6 Matching 6.. Matching poses Association of anchoring points for two simiar poses can be shifted to neighboring anchoring points [7]. Hence, we need to deveop a distance metric which may aow the region to shift at most one position without any penaty and cacuate the distance between poses

4 effectivey. This can be done by constructing a matrix PM according to equations (9) and (0) and then appying agorithm 6.. PM i, j r dist v i, v j i j or i j 3 otherwise where i, j= to4, v and v are two pose vectors. r v i, dist v j 6 v ( i) v v ( i) v 0.5 v ( i) v 0.5 & v ( i) v Otherwise 0.5 where is the index variabe of 6 features for each region with orientation being v ( i) or v ( j. ) Agorithm 6.: Function r_dist(pm) Begin pose_d=0; m_ee=min(pm); whie m_eeinf [r c]=position(m_ee,pm); pose_d=pose_d+m_ee; assign INF to a eements of row r in PM assign INF to a eements of coumn c in PM if rc // anchor point is shifted pose_d=pose_d+pm(c,r); assign INF to a eements of row c in PM assign INF to a eements of coumn r in PM end m_ee=min(pm); end return pose_d end where INF=, MIN(X) returns the minimum eement in matrix X and POSITION(j,X) returns the ocation of j inside matrix X in terms of row and coumn. If there are mutipe j, return the ocation of j with owest row and coumn vaues. 6.. Matching sequences We empoyed DTW agorithm to match mode with input sequence, since DTW not ony find the simiarity of two sequences regardess of non-inear variation in the time dimension, aso it checs the order of poses between two sequences. In our earier wor, we showed sequence matching where ony one type of action in mode data but in current situation there can be mutipe types of action in mode data. Hence, we need to extend our earier idea to sequentia matching of DTW for mutipe action types. Fig. 4 shows the one iteration (querying one action type) matching procedure. Suppose we have a query ie action foowed by action foowed by action q. Let action have p number of mode sequences. Thus, p DTW matrices are constructed (Fig. 4) by using the recurrence (9) (0) reation of equation () and the same DTW constraints (one cyce of mode aong the rows, whoe input sequence aong the coumns, reaxed end point, sope constraints) as our earier wor [7]. D i, j wv i, j D i, j d ( i, j) min D ( i, j ) D ( i, j ) wh i, j () D, t d (, t) ini t, D i,, ini p _ row, _ row j D, min n, j sp, j row 0,0,0,..., _ m 0 0 where i= to n, j= to m, = to q, t= to m, = to p, n and m are the number of poses in mode and input sequences respectivey, q is the number of query action type, p is the number of mode sequences for query action, d (i,j) distance between pose i of mode and input pose j, D is the DTW matrix for mode, w h and w v are the sope constraints: w h(v) =cons_m h(v) if cons_m h(v) > otherwise 0 (cons_m h(v) is consecutive move of warping path in horizonta (vertica) direction), sp(,j) is the difference between speed of mode and the speed of input poses r to j where r is the starting pose of a warping path which ends at pose j of input sequence. Fig. 4: Matching process for one query action type, action. D is the DTW matrix for mode, sp(,j) is the speed difference between mode and input sequence.

5 wa-carry run-carry stand-reach stand-wave crouch-run wa-stand stand-picup run-picup wa-jump crouch-jump-run run-picup-run wa-jump-carry wa-picup-carry wa-stand-run wa-stand-wave-wa (a) ran Fig. 5: Searching resut of 5 queries. (a) Raning of reevant sequences by our system. Queries are shown on eft. (b) Comparison with Iizer et a. [5] method. Queries are numbered according to their position in (a), e.g. query wa-carry is and wa-stand-wave-wa is 5. Average precision query (b) Next _row vector is formed by taing minimum of a the ast row of p DTW matrices. This _row vector is then used as ini vector for next iteration (equation ()). After performing matching with a the query action types, fina matching score is cacuated as m m _ score min j _ row 7 Experimenta resuts q j Our system is evauated by a pubicy avaiabe dataset UIUC compex activity dataset [5]. This data set contains 73 video sequences where in each sequence an actor performing mutipe actions sequentiay. There are 5 actors wearing 5 different outfits. Iizar et a. [5] used a the sequences of actor as the training and rest sequences as the testing (input) data. We aso empoyed the same settings. To create the mode data, each training video sequence is divided into sub-sequences so that each sub-sequence contain ony one cyce of one action type. Performance over a set of queries is evauated by mean average precision (MAP), which is the area under precision reca curve of a query. Mathematicay, average precision P avg over a dataset, S is defined as N p r re r r (3) P avg number of reevant sequence in S () where, r is the ran of a video sequence (owest distance video is ran ) re(r) is binary reevance vector (reevant=, otherwise 0), p(r) is the precision at ran r and N is the number of retrieved sequence. For mutipe queries mean vaue of P avg of a queries is MAP vaue. Iizer et a. [5] performed 5 queries on whoe dataset where the queries are shown in Fig. 5(a). We aso performed these queries and showed the raning of reevant sequences (sequences contain a the query action type in the same order as the query) in Fig. 5(a). Among 5 queries, 9 queries give higher average precision than [5] (Fig. 5(b)) which impies that our system can search for compex activity in a dataset effectivey. Iizer [5] method is a tracer based method which faced difficuties due to sef-occusion whereas in our method sefoccusions is not a probem. Main chaenge faced by our system is insufficient mode data (for some action type, ony one mode sequence). If more training data are provided, our system efficiency woud be improved. 8 Concusion In this paper, a sihouette based method is proposed where features are extracted from the surrounding regions (negative space) of the sihouette to search compex human activity in a singe video sequence. Main contribution of this wor is to extend the idea of negative space to search mutipe action types sequentiay in a video sequence by eeping a the advantages of negative space approaches. Negative space based methods have severa advantages: robust to noise, shadow, partia occusions, variations of cothing with simpe representation of pose. Our system performance, which is better than the state-of-art method, can be further improved by incuding more training data in the system. However, the system is not fuy invariant to viewing ange which can be overcome by incorporating different viewing ange data in the training set. 9 References [] Poppe, R.: A Survey on Vision-based Human Action Recognition, Image and Vision Computing, 00, 8, (6), pp [] Wang, L., Hu, W., and Tan, T.: Recent Deveopments in Human Motion Anaysis, Pattern Recognition, 003, 36, (3), pp

6 [3] Godenberg, R., Kimme, R., Rivin, E., and Rudzsy, M.: Behavior Cassification by Eigen Decomposition of Periodic Motions, Pattern Recognition, 005, 38, (7), pp [4] Diaf, A., Ksantini, R., Boufama, B., and Benamri, R.: A Nove Human Motion Recognition Method Based on Eigenspace. Proc. Lecture Notes in Computer Science, 00, pp [5] Iizer, N., and Forsyth, D.A.: Searching for Compex Human Activities with No Visua Exampes, Int. J. Comput. Vision, 008, 80, (3), pp [6] Wang, X., Ma, X., and Grimson, W.E.: Unsupervised Activity Perception in Crowded and Compicated Scenes Using Hierarchica Bayesian Modes, IEEE Trans. Pattern Ana. Mach. Inte., 009, 3, (3), pp [7] Rahman, S.A., Li, L., and Leung, M.K.H.: Human Action Recognition by Negative Space Anaysis. Proc. Cyberwords (CW), 00, pp [8] Li, L., Huang, W., Gu, I.Y.-H., and Qi, T.: Statistica Modeing of Compex Bacgrounds for Foreground Object Detection, Image Processing, 004, 3, (), pp [9] Teague, M.R.: Image Anaysis via the Genera Theory of Moments, Journa of the Optica Society of America, 980, 70, pp