Utility-based Camera Assignment in a Video Network: A Game Theoretic Framework

Transcription

1 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK Utiity-based Camera Assignment in a Video Network: A Game Theoretic Framework Yiming Li, Student Member IEEE and Bir Bhanu, Feow IEEE Abstract In this paper an approach for camera assignment and handoff in a video network based on a set of user-suppied criteria is proposed. The approach is based on game theory, where bargaining mechanisms are considered for coaborations as we as for resoving conficts among the avaiabe cameras. Camera utiities and person utiities are computed based on a set of user-suppied criteria, which are used in the process of deveoping the bargaining mechanisms. Different criteria and their combination are compared with each other to understand their effect on camera assignment. Experiments for muti-camera muti-person cases are provided to corroborate the proposed approach. Intuitive evauation measures are used to evauate the performance of the system in rea-word scenarios. The proposed approach is aso compared with two recent approaches based on different principes. The experimenta resuts show that the proposed approach is computationay more efficient, more robust and more fexibe in deaing with the user-suppied criteria. Index Terms Bargaining Mechanism, Camera handoff, Dynamic Camera Seection D I. INTRODUCTION UE to the broad coverage of an environment and the possibiity of coordination among different cameras, video sensor networks have attracted much interest in recent years. Athough the fied-of-view (FOV) of a singe camera is imited and cameras may have overapping/non-overapping FOVs, seamess tracking of moving objects is desired. This requires two capabiities: camera assignment and camera handoff, which are the subjects of this paper. We define camera assignment as a camera-object map, which tes us at each time instant which camera is being used to foow which object. Camera handoff is a dynamic process that the system transfers the right of tracking an object from one camera to another without osing the object in the network. The avaiabiity of camera handoff capabiity wi provide the Manuscript submitted January 5, 200, revised March 30, 200, accepted May 03, 200. This work was partiay supported by NSF grants 05574, , and ONR grants (DoD Instrumentation and the Aware Buiding). The contents of the information do not refect the position or poicy of the US Government. Yiming Li is with the Center for Research in Inteigent Systems, Eectrica Engineering Department, University of Caifornia at Riverside, Riverside, CA 9252 (e-mai: yimi@visab.ee.ucr.edu). Bir Bhanu is with the Center for Research in Inteigent Systems, Eectrica Engineering Department, University of Caifornia at Riverside, Riverside, CA 9252 (e-mai: bhanu@cris.ucr.edu). much needed situation assessment of the environment under surveiance. It is cear that the manua camera handoff wi become unmanageabe when the number of camera is arge. In addition, it is unreaistic to dispay and manuay monitor the surveiance videos captured from a arge number of cameras simutaneousy. Therefore, we need to deveop surveiance systems that can automaticay perform the camera assignment and handoff tasks, and then adapt to the appropriate video streams avaiabe from the currenty used cameras. In this paper, we provide a new perspective to the camera handoff probem based on game theory. The merit of our approach is that it is independent of the camera topoogy. When mutipe cameras are used for tracking and where mutipe cameras can see the same object, the agorithm can automaticay provide an optima as we as stabe soution of the camera assignment. Since game theoretic approach aows deaing with mutipe criteria optimization, we are abe to choose the best camera based on mutipe criteria that are seected a priori. The detaied camera caibration or 3D scene understanding is not needed in our approach. In the rest of this paper,section II describes the reated work and contributions of this work. Section III formuates the camera assignment and handoff probem and then constructs the utiities and bargaining steps. Section IV discusses the impementation of this approach and shows the experimenta resuts. Comparison resuts of the proposed approach with two recent approaches are aso shown. Finay, Section V concudes the paper. II. RELATED WORK AND OUR CONTRIBUTION A. Reated Work There have been many papers discussing approaches for doing camera assignments in a video network. The traditiona approaches generay fa into two categories: topoogy-based and statistics-based. The approaches beonging to the first category [-4] rey on the geometrica reationships among cameras. These reationships tend to become quite compicated when the topoogy becomes compex and it is difficut to earn the topoogy based on the random traffic patterns [5]. The approaches beonging to the second category [6-0] usuay depend on the objects trajectories, whie other factors such as orientation, shape, face etc., which are aso very important for visua surveiance, are not considered. A comparison of reated work with the proposed approach is summarized in Tabe I.

2 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 2 TABLE I COMPARISON OF THIS PAPER WITH THE RELATED WORK Author Approach Comments Javed et a. [] Park et a. [2] Jo & Han [3] Qureshi & Terzopouos [4] Kettnaker & Zabih [6] Chang & Gong [7] Kim et a. [8] Cai & Aggarwa [2] Tessens et a. [4] This paper Find the imits of overapping FOVs of mutipe cameras. Cameras are chosen based on the distance between the person and the edge of FOV. Create distributed ook-up tabes according to how we the cameras can image a specific ocation. Construct a handoff function by computing the co-occurrence to occurrence ratio for seected pairs of points in the FOVs of two cameras. Modes conficts that may arise during camera assignment as a constraints satisfaction probem; each camera assignment that passes the hard constraints is assigned a weight. The camera assignment with the highest weight is seected. Choose the object trajectory with the highest posterior probabiity. Use Bayesian modaity fusion. A Bayesian network is used to combine mutipe modaities for matching subjects among mutipe cameras. Find a dominant camera for an object based on the ratio of the numbers of bocks on the object to that in the camera FOV. Matches human objects by using mutivariate Gaussian modes. Messages reated to a distributed process are sent to a base station to determine the principa camera based on a score which is decided by experiments offine. Modes the camera assignment probem as a potentia game using the vehice-target mode. Uses bargaining mechanism to obtain a converged soution with a sma number of iterations. Significant increase in computation with the increase in the number of cameras and persons. Based ony on singe criterion; depends on the information that may not be avaiabe for each network. Computation cost is high for increased handoff resoution. Handoff ambiguity and faiure wi arise when there are a arge number of cameras or persons. A eader node is required. One has to cacuate a the soutions that can pass the constraints and seect the best one. The ranking procedure can be time consuming as the probem becomes more compicated. Camera assignment is conducted based on the paths of the objects. Fronta view can be ost even when it is avaiabe. Geometry/recognition based, hard to combine the two together for a decision; mainy focuses on the occusion probem, not other criteria for camera assignment. The dominant camera has to be aided by views from other cameras. Ony size and ange are taken into account. Camera caibration is needed. Use both distributed and centraized contro. The criteria are determined offine so it is not suitabe for active camera seection and contro. The principa view is compemented by heper views. Can decide the camera assignment based on mutipe criteria; no camera caibration is needed; independent of the camera network topoogy. B. Contributions of This Paper Our approach differs from the conventiona approaches [- 4, 6-8, 2, 4], shown in Tabe, in the foowing key aspects:. Game Theoretic Approach: We propose a game theoretic approach for camera assignment and handoff probem using the vehice-target mode [3].We mode the probem as a muti-payer potentia game and aow for both coordination and conficts among the payers. 2. Mutipe Criteria for Tracking: Mutipe criteria are used in the design of utiity functions for the objects being tracked. The equiibrium of the game provides the soution of the camera assignment. The bargaining mechanism makes sure that we can get a stabe soution, which is optima or near optima, after ony a sma number of iterations [3]. 3. Best Camera Seection: We do not use the traditiona master-save system [4]. Instead, by seecting the best camera, we can have a good enough view, based on the user-suppied criteria, for observation of some specific target and simutaneousy free the other cameras in the network for other tasks. Thus, the system can perform the tracking task with a minimum number of cameras, or, can perform more tasks with the same number of cameras. 4. Experimenta Resuts: Unike some of the previous work [4], we evauate the proposed approach in the context of rea network using rea data and show promising resuts. III. TECHNICAL APPROACH A. Motivation and Probem Formuation Game theory is we known for anayzing the interactions as we as conficts among mutipe agents [5, 6]. Anaogousy, in a video sensor network, coaborations as we as competitions among cameras exist simutaneousy. The cooperation ies in the fact that a the avaiabe cameras, those which can see the target person, have to coaborate to track the person so that the person can be foowed as ong as possibe. On the other hand, the avaiabe cameras aso compete with each other for the rights of tracking this person, so that a camera can maximize its own utiity, as a camera s utiity is cosey reated to how we it can track a person. This enightens us to view the camera assignment probem in a game theoretic manner. A game is the interactive process [7] among a the participants (payers) of a game, who strive to maximize their utiities. The utiity of a payer refers to the wefare that the payers can get in the game. In our probem, for each person to be tracked, there exists a muti-payer game, with the avaiabe cameras being the payers. If there are mutipe persons in the system, this becomes a mutipe of muti-payer game being payed simutaneousy [8]. Vehice-target assignment [3] is a cassica muti-payer game that aims to aocate a set of vehices to a group of targets and achieves an optima assignment. Viewing the persons being tracked as vehices whie the cameras as targets, we can adopt the vehice-target assignment mode to choose the best camera for each person. In the foowing, we propose a game theory based approach that is we suited to the task at hand. B. Game Theoretic Framework Game theory invoves utiity, the amount of wefare an agent derives in a game. We are concerned with three utiities: ) Goba utiity: the overa degree of satisfaction for tracking

3 ... This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 3 Bargaining among avaiabe cameras Mutipe video streams Detecting and tracking objects in each camera (incuding face detection) User-suppied criteria Criterion : size Criterion 2: position Criterion 3: view Computation of camera utiities Updating Assignment probabiities Computation of person utiities Prediction Estimated person utiities performance. 2) Camera utiity: how we a camera is tracking persons assigned to it. 3) Person utiity: how we a person is satisfied whie being tracked by some camera. Our objective is to maximize the goba utiity whie making sure that each person is tracked by the best camera. When competing with other avaiabe cameras, the cameras bargain with each other. Finay a decision is made for the camera assignment based on a set of probabiities. An overview of the approach is iustrated in Fig.. Moving objects are detected in mutipe video streams. Their properties, such as the size of the minimum bounding rectange and other region properties (coor, shape, ocation within FOV etc.) are computed. Various utiities are cacuated based on the user-suppied criteria and bargaining processes among avaiabe cameras are executed based on the prediction of person utiities from the previous iteration step. The resuts obtained from the strategy execution are in turn used for updating the camera utiities and the person utiities unti the strategies converge. Finay those cameras with the highest converged probabiities are used for tracking. This assignment of persons to the best cameras eads to the soution of the handoff probem in mutipe video streams. A set of key symbos and their notations used in the foowing discussion are given in Tabe II. TABLE II NOTATIONS OF SYMBOLS USED IN THE PAPER Symbos Notations P i Person i C j Camera j N P Tota number of persons in the entire network at a given time Tota number of cameras in the network at a given time N C A i n C The set of cameras that can see P i, A i = {A i, A 2 n i,, A C i } Number of cameras that can see person i, number of eements in A i n P Number of persons currenty assigned to camera C j a i The currenty assigned best camera for person i a i The assignment of cameras for the persons excuding P i a The assignment of cameras for a persons, a = (a i, a i ) U Cj (a) Camera utiity for camera j U Pi (a) Person utiity for person i U g (a) Goba utiity U Pi (k) Predicted person utiity for person i at step k, U Pi (k) = n U Pi k,, U Pi k,, U C T Pi k, where UPi k is the predicted person utiity for P i if camera a is used p i k Probabiity of person i s assignment at step k, p i k = n p i k,, p i k,, p C T i k, where pi k is the probabiity for camera a to track person P i B.. Computation of Utiities We define the foowing properties of our system: Fina camera assignment and hand-off Fig.. Game theoretic framework for camera assignment and hand-off.. A person P i can be in the FOV of mutipe cameras. The avaiabe cameras for P i beong to the set A i. C 0 is a virtua camera that does not actuay exist. We assume a virtua camera C 0 is assigned to P i when there is no rea camera in the network avaiabe to track P i. 2. A person can ony be assigned to one camera. The assigned camera for P i is named as a i. 3. Each camera can be used for tracking mutipe persons. We use a to denote the camera assignment for a the persons, and a i denotes the assigned camera for P i. For P i, when we change the camera assignment from a i to a i whie assignments for other persons remain the same, if we have U Pi a i, a i < U Pi a i, a i U g a i, a i < U g a i, a i () the person utiity U Pi is said to be aigned with the goba utiity U g, where a i stands for the assignments for persons other than P i, i.e., a i = (a,, a i, a i+,, a NP ). So, the camera assignment resut a can aso be expressed as a = a i, a i. We define the goba utiity as U g a = C j C U Cj (a) (2) where U Cj (a) is the camera utiity and defined to be the utiity generated by a the engagements of persons with a particuar camera C j. Now, we define the person utiity as U Pi a = U g a i, a i U g C 0, a i = U Cj a i, a i U Cj C 0, a i (3) where, C 0 is a virtua camera. The person utiity U Pi (a) can be viewed as a margina contribution of P i to the goba utiity. To cacuate (3), we have to construct a scheme to cacuate the camera utiity U Cj (a). We assume that there are N Crt criteria to evauate the quaity of a camera used for tracking an object. Thus, the camera utiity can be buit as U Cj a i, a i = Crt s n P N Crt s= = (4) where n P is the number of persons that are currenty assigned to camera C j for tracking and Crt are the criteria that are suppied by the user. Pugging (4) into (3) we can obtain N Crt n P n P = ) (5) s P i U Pi a i, a i = ( s= Crt s s= Crt s where s P i means that we excude person P i from the those who are being tracked by Camera C j. One thing to be noticed here is that when designing the criteria, we have to normaize

4 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 4 them. Besides this requirement, it does not matter what kind of criteria is used to be fed into the bargaining mechanism which is discussed beow. B.2. Criteria for Camera Assignment and Handoff The choice of a criterion to be used for camera assignment and handoff depends on the users requirements. There might be different criteria for different appications, such as criteria for power consumption, time deay, image resoution etc. The camera assignment resuts may change due to appying different criteria. Our goa is to find the proper camera assignment soution quicky based on whatever criteria are suppied by the user. In the foowing, we provide four criteria, which incude human biometrics, which can be used for camera assignment and handoff. Criterion : The size of the tracked person. It is measured by the ratio of the number of pixes inside the bounding box of the person to the size of the image. That is r = # of pixes inside te bounding box # of pixes in te image pane Here, we assume that neither a too arge nor a too sma object is convenient for observation. Assume that λ is the threshod for best observation, i.e., when r = λ this criterion reaches its optima vaue. Crt s = r, wen r < λ λ r, wen r λ (6) λ where λ 0, is defined as the optima ratio of the size of the minimum bounding box for the human body to the size of the image. These two sizes can be obtained by reading the coordinates of the bounding box and the size of the image. λ is dependent on the orientation of the camera and the ocation of a region-of-interest (ROI) in the image pane. Ony in extreme rare situations a ROI wi have a minimum width of one pixe. The vaue of λ remains vaid at a times. Because of these reasons we do not do any camera caibration to find its extrinsic or intrinsic parameters. An exampe for the function Crt s when λ = is shown in Fig. 2 as an iustration. 5 Fig. 2. Function of Crt s when λ = Criterion 2: The position of a person in the FOV of a camera. It is measured by the Eucidean distance that a person is away from the center of the image pane Crt s2 = (x x c ) 2 +(y y c ) 2 2 x c 2 +y c 2 (7) where (x, y) is the current position (body centroid) of the person and (x c, y c ) is the center of the image. 5. Criterion 3: The view of a person. It is measured by the ratio of the number of pixes on the detected face to that of the whoe bounding box. That is R = # of pixes on te face # of pixes on te entire body We assume that the threshod for the best fronta view is ξ, i.e., when R = ξ (ξ (0,)) the view of the person is the best. Crt s3 = r, wen R < ξ ξ R, wen R ξ (8) ξ Criteirion 4: Combination of criterion (), (2) and (3). It is given by the foowing equation, Crt s4 = 3 = w Crt s (9) where w is the weight for different criterion. It is to be noticed that a these criteria are appropriatey normaized for cacuating the corresponding camera utiities. B.3. Bargaining Among Cameras As stated previousy, our goa is to optimize each person s utiity as we as the goba utiity. Competition among cameras finay eads to the Nash equiibrium [2], as the soution of the camera assignment and handoff. Unfortunatey, this Nash equiibrium may not be unique. Some of the soutions may not stabe, which are not desired. To sove this probem, a bargaining mechanism among cameras is introduced, to make these cameras finay come to a compromise and generate a stabe soution. When bargaining, the assignment in the k t step is made according to a set of probabiities p i k = p i k,, p i k,, p i n C k T where n C is the number of cameras that can see the person n P i and C p = i (k) =, with each 0 p i k, =,, n C. We can generaize p i (k) to be p i k = p i k,, p i k,, p i N C k T by assigning a zero probabiity for those cameras which cannot see the person P i, meaning that those cameras wi not be assigned according to their probabiity. Thus, we can construct an N C N P probabiity matrix p (k) p NP (k) N p C (k) N p C NP (k) At each bargaining step, we wi assign a person to the camera which has the highest probabiity. We assume that one camera has no information of other cameras utiities at the current step, which makes it hard to cacuate a the possibe current person utiities. So, we introduce the concept of predicted person utiity U Pi (k): Before we decide the fina assignment profie, we predict the person utiity using the previous person s utiity information in the bargaining steps. As shown in (5), person utiity depends on the camera utiity, so, we predict the person utiity for every possibe camera that

5 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 5 may be assigned to track it. Each eement in U Pi (k) is cacuated by (0) U Pi k + = U Pi k + (U p i k P i a k U Pi k ), a i k = A i U Pi k, oterwise (0) with the initia state U Pi () to be assigned arbitrariy as ong as it is within the reasonabe range for U Pi (k), for =,, n C. For the symbos used in Equation 0, note that A i is the th camera that is in the set of avaiabe cameras for person P i, which is different from C, the th camera in the system. C can be in more than one avaiabe camera sets for different persons, whie A i is the th component in A i, the set of avaiabe cameras for person P i. It means that A i is unique in the set for person P i. Once these predicted person utiities are cacuated, it can be proved that the equiibrium for the strategies ies in the probabiity distribution that maximizes its perturbed predicted utiity [0], p i (k) U Pi k + τh(p i (k)) () where H p i k = p i (k) og (p i k ) (2) is the entropy function and τ is a positive parameter beonging to [0,] that contros the extent of randomization, where og means taking the og of every eement of the coumn vector p i k and resuting in a coumn vector. The arger τ is, the faster the bargaining process converges; the smaer the τ is, the more accurate resut we can get. So, there is a tradeoff when seecting the vaue of τ. We seect τ, empiricay, as 0.5 in our experiments. The soution of () is proved [0] to be p i k = e τ U P k i e τ U P k i + +e τ U n C P k i (3) After severa steps of cacuation, the resut of p i (k) tends to converge. Thus, we finay get the stabe soution, which is proved to be at east suboptima [3]. B.4. Game Theoretic Agorithm This overa agorithm is summarized in Agorithm. Agorithm : Game theoretic camera assignment and handoff Input: Mutipe video streams. Output: A probabiity matrix for camera assignments are made. Agorithm Description: At a given time, perform motion detection and get the seected properties for each person that is to be tracked. For each person and each camera, decide which cameras can see a given person P i. For those which can see the person P i, initiaized the predicted person utiity vector U Pi (). Repeat. Compute the Crt s for each avaiabe camera. 2. Compute the camera utiities U Cj (a) by (4). 3. Compute the person utiities U Pi (a) by (5). 4. Compute the predicted person utiities U Pi (k) by (0). 5. Derive the strategy by p i (k) using (3). Unti The strategies for camera assignments converge. Do camera assignment and handoff based on the converged strategies. The bargaining mechanism and the criteria are tighty integrated in the proposed game theoretic approach. The bargaining process is based on a set of criteria, since the utiities used to update in each bargaining step are cacuated using these criteria. Note that different criteria impy different emphasis and the definition of error (see Section IV C) depends on them. A. Data and Parameters IV. EXPERIMENTAL RESULTS A.. Data In our experiments, we tested the proposed approach on five cases: () 3 cameras, person, (2) 3 cameras, 2 persons, (3) 2 cameras, 3 persons, (4) 4 cameras, 4 persons and (5) 4 cameras, 6 persons. These experiments incude from the simpe case, 3 cameras, person, to a compicated case, 4 cameras, 6 persons. There are both cases with more peope than cameras (see Fig. 4) and more cameras than peope (see Fig. 6 and Fig. 0), which show that the performance of the proposed approach wi not be infuenced by reative numbers of cameras and persons. Both indoor and outdoor experiments are provided. The engths of the video sequences vary from 450 frames to 700 frames. The frame rate for a indoor videos is 30 fps whie that for outdoor videos is 5 fps. The cameras used in our experiments are a Axis 25 PTZ cameras, which are paced arbitrariy. To fuy test whether the proposed approach can hep to seect the best camera based on the user suppied criteria, some of the FOVs of these cameras are aowed to interact whie some of them are non-overapping. The experiments are carried out in three different paces with no camera caibration done before hand. The trajectories are randomy chosen by the persons for waking. We visuaize the camera configuration and the persons trajectories for the 5 cases in Fig. 3. A.2. Parameters In our experiments, we empiricay give vaues to the parameters required by the criteria introduced in Section III B.. λ = 5, ξ = 6, w = 0.2, w 2 = 0., w 3 = 0.7. These parameters are hed constant for a the experiments reported in this paper. B. Tracking and Face Detection B.. Tracking A the experiments are conducted using the Continuous Adaptive Meanshift (Camshift) tracker [9] to evauate the camera seection and handoff mechanisms. Theoreticay, which tracker is used is not important as ong as it can provide the tracking information that consists of size (size of the bounding box of a person) and ocation (position of the centroid of the bounding box) of a person. It is to be noticed that the same tracker is used for a the experiments and a the camera assignment approaches that are compared to fiter out the infuence of a tracker to the camera seection resuts. The waking persons are initiay seected by an observer manuay when a person enters the FOV of a camera as detected by the background subtraction method. The persons who participated in the experiments wear cothes in distinct

6 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 6 3 cameras, person 3 cameras, 2 persons 2 cameras, 3 persons 4 cameras, 4 persons 4 cameras, 6 persons Fig. 3. Camera configuration and the persons trajectories in the experimented cases. coors, so different persons can be identified by cacuating the correation of the hue histograms of the pixes inside their bounding boxes (ROIs) using the function CompareHist [20]. Errors Caused by the Tracker (N c = 2, N p = 3, Indoor) There are some errors that are caused by the faiure of the tracker. In Fig. 4, we show some error frames in a 2 cameras, 3 persons case, which are due to the faiure of the Camshift tracker. The Camshift tracker is not robust when severe occusion happens and it can be distracted by the object with simiar coors as the target. However, the camera assignment resuts are correct if we ignore the errors that are caused by the tracker, i.e., if we assume that the tracker provides a correct ROI for the target, then the camera assignments, performed based on the user-suppied criteria, are correct. For instance, in Fig. 4 (4- and 4-2), the system shoud seect camera to track the person in red, where the person has a fronta view which is preferred according to the user-suppied criteria. However, the tracker for the person in red is distracted by the red piow, which causes error for the camera assignment. But if we assume that the ROIs returned by the tracker are correct, then based on the size and position of the person in red (fronta face is not avaiabe because of the wrong tracking resut), the system seects the correct camera Camera Camera 2 Fig. 4. An exampe for the faiure of the Continuous Adaptive Meanshift (Camshift) tracker. We ony draw ROIs in the cameras that are seected to track the persons. We generate bounding boxes ony for those cameras that are seected to track a person. B.2. Face Detection Face detection is done in a particuar region (top /3 height of the ROI), provided by the tracker. We use the cascade of haar feature based cassifiers for face detection [20]. It can detect faces correcty in 90%+ cases when the tracker returns a correct ROI. C. Performance Measures In our experiments, the bottom ine is to track waking persons seamessy, i.e., the system wi foow a person as ong as the person appears in the FOV of at east one camera. In the case where more than one camera can see the persons, we assume that the camera that can see the person s face is preferabe. This is because in surveiance systems, the fronta view of a person can provide us more interesting information than other views. So, based on this criterion, we define the camera assignment error in our experiments as: () faiing to track a person, i.e., a person can be seen in some cameras in the system but there is no camera assigned to track the person, or (2) faiing to get the fronta-view of a person whenever it is avaiabe. We define these error terms in the foowing: N ost - the number of times that a target person is ost. It is determined if the bounding box returned by the tracker covers ess than 30% of the person s actua size or is arger than 50% of the person s actua size during tracking. The term region-of-interest (ROI) and bounding box are used interchangeaby in this paper. N fv - the number of times a fronta view is detected but not seected by any camera. Note that in our experiments, there is no case where a fronta view is detected but the person is ost during tracking. So, the intersection of the above two cases shoud be empty, i.e. N ost N fv =. NP i C j - the number of persons appearing in Camera C j in frame i. N f the tota number of frames in an experimented video. NPC i - the number of cameras with no persons in frame i. N C - the tota number of cameras in an experiment. The tota error of a video is defined as Err = N ost + N fv (4) The error rate is defined as the error normaized by tota the numbers of cameras and persons in a frames. Frames in which there are no persons are counted as correct frames, since there are no errors caused by osing a person or ose the fronta view of a person. Frames with more than one person in the FOV of a camera are mutipy counted to normaize by the mutipe persons. Error rate ER is defined as ER = N f i= Err NC C j N f NP j = i + NPC i= i NP i C j =0 D. Evauation of Game Theoretic Framework (5) D.. Experiment #: Criterion Seection (N C = 3, N P = 2, Indoor) Since there are mutipe criteria to be used in the experiments, we first test the performance for different

7 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 7 criterion in a 3 cameras, 2 persons case. A genera description of the videos is shown in Tabe III. TABLE III EXPERIMENT #. OVERVIEW OF VIDEOS FOR EACH CAMERA AND THE NUMBER OF HANDOFFS THAT ARE TAKEN PLACE (NOF: NUMBER OF FRAMES) Nof ( 0 person) Nof ( person) Nof ( 2 persons) Nof (with occusion) No. of handoffs (Crt s4 ) Cam Cam Cam Different experiments are carried out using the singe and the combined criterion described in Section III B.. Some typica resuts are shown in Fig. 5. To make it convenient for a comparison, we show the tracking resuts for other cameras as we, no matter whether they are seected for tracking or not. The cameras, for which the bounding boxes are drawn in bue, are seected for tracking whie the ones in red or green are not as good as the bue ones. may fai to track a person when no face can be detected. As shown in Fig. 5(f), athough the person is in the FOV of some camera, the person is ost based on criterion 3. So, finay, we come up with a weighted combination of these three criteria. As stated previousy, we use 0.2, 0. and 0.7 as the weights for these three criteria respectivey so that, in most cases, the system wi choose the camera which can see a person s face. For those frames where there is person without the detected face, the combination criterion can aso provide the best camera based on criteria and 2 and, thus, reaizing continuous tracking. A the camera handoffs, when appying the combined criterion, are shown in Fig. 6. The error rate in this case is 5.56%, whie that for using criterion to 3 ony are 25.56%, 0.00% and 30.00%, respectivey. (a) (b) (c) (a) (b) (c) (d) (e) (f) Camera Camera 2 Camera 3 Fig. 5. Experiment #. A comparison for using different criteria. The first row and the second row are for two time instants respectivey. The first coumn through the third coumn are using criterion to criterion 3, respectivey. Fig. 5 (a) to (c) use criterion to 3 at time instant whie (d) to (f) use criterion to 3 at time instant 2. It can be obseved from Fig. 5 (d) that the probem for using criterion ony is that when the persons are getting cose to the cameras, the size of the bounding box increases, and whie the resoution is not very high, persons are not cear enough. Meanwhie, there are cases such that when a person is entering the FOV of a camera, the size of the person is not sma but ony part of the body is visibe. This shoud not be preferred if other cameras can give a better view of the body. Thus, we introduced criterion 2, considering the reative position of persons in the FOVs of the cameras. The coser the centroid of a person is to the center of the FOV of a camera, the higher the camera utiity is generated. We can observe that when appying criterion 2 in Fig. 5 (e), the camera with the person near the center is chosen and we can obtain a higher resoution of the person compared to the resuts based on criterion in Fig. 5 (d). However, the probem for using criterion or criterion 2 ony is that we reject the camera(s) which can see a person s face, which is of genera interest. This case is shown in Fig. 5 (a) (b) and (d). To sove this probem, we deveoped criterion 3 (the view of the person). So, when appying criterion 3, we obtain a more desirabe camera with a fronta view of the person in Fig. 5 (c) and (f). Whereas criterion 3 can successfuy seect a camera with a fronta-view person, it (d) (g) (h) (i) Camera Camera 2 Camera 3 Fig. 6. Experiment #. A camera handoffs when appying the combined criterion for 3 cameras, 2persons case. The cameras that are seected for tracking a person provides a bue bounding box for that person, otherwise it provides green bounding box for the person in red and red bounding box for the person in green. The number of handoffs in this 3 cameras, 2 persons case is give in Tabe III. Camera utiities, person utiities and the corresponding assignment probabiities for the using the combined criterion is shown in Fig. 7, where Probabiity[i][j] stands for the probabiity that C j is assigned to track P i. Fig. 7. Experiment #. Utiities and assignment probabiities for each processed frame when using the combined criterion. We use the combined criterion for a the other experiments (e) (f)

8 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 8 in the rest of this paper. D.2. Convergence of Resuts for Bargaining For the above experiments, in most cases, the probabiities for making the assignment profie converges (with ε < 0.05, where ε is the difference between the two successive resuts) within 5 iteration. So we use 5 as the number of iterations threshod when bargaining. Thus, for those cases that wi not converge within 5 iterations, there may be an assignment error based on the unconverged probabiities. In Fig. 8 we pot the number of iteration with respect to every processed frame for Experiment #. It turns out that the average number of iterations is.37. As the numbers of persons and cameras increase, this bargaining system wi save a ot of computationa cost to get the optima camera assignments. A typica convergence for one of the assignment probabiities in a bargaining among cameras is given in Fig. 9. We aso show an exampe of error caused by the faiure of the bargaining mechanism in a more compicated (4 cameras, 6 persons) Experiment #6 discussed ater in the comparison part. Fig. 8. Experiment #. Number of iteration for the bargaining mechanism in each frame. at x. The mean probabiity that moving objects are simutaneousy detected at x in the FOV of one camera and x in the FOV of another camera is caed a co-occurrence of x and x. The COR approach decides whether two points are in correspondence with each other by cacuating the cooccurrence to occurrence ratio. If the COR is higher than some predefined threshod, then the two points are decided to be in correspondence with each other. When one point is getting cose to the edge of the FOV of one camera, the system wi handoff to another camera that has its corresponding point. However, the COR approach in [3] has been appied to two cameras ony. We generaize this approach to the cases with more cameras by comparing the accumuated COR in the FOVs of mutipe cameras. We randomy seect 00 points on the detected person, train the system for 0 frames to construct the correspondence for these 00 points, cacuate the cumuative CORs in the FOVs of different cameras and seect the one with the highest vaue for handoff. Experiments have been done to compare the COR approach with our approach for the 3 cameras, person case (Experiment #2) and the 3 cameras, 2 persons case (Experiment #3). Experiment #2: Comparison with COR Approach (N C = 3, N P =, Indoor) The handoff process by using the COR approach and the corresponding frames by using our approach (may not be the handoff frames) are shown in Fig. 0. In Fig. 0 (g) to (h), the COR approach switches to camera, whie our proposed approach sticks to camera 2 (Fig. 0 (c) to (d)) to get the fronta view of the person. The COR approach needs a time period to construct the correspondence between different views. We et this period to be 0 frames. As a resut, there is some time deay for the handoff. For instance, in Fig. 0 (a) to (b), our approach has aready seected camera 3 in (a), where a fronta view of the person is aready avaiabe and the size of the person is acceptabe, whie the COR approach switched to camera 3 in (d) when the person is detected as eaving the FOV of camera 2 and entering the FOV of camera 3. Fig. 9. Experiment #. A typica convergence in the bargaining process (Frame 56, camera 2, for the person in green). (a) (b) (c) (d) E. Comparison of Game Theoretic Approach with Other Reated Approaches In this section, we wi compare our approach with two other approaches: the first approach [3] performs camera handoff by cacuating the co-occurrence to occurrence ratio (COR). We wi ca this the COR approach. The second approach performs the camera assignment probem by soving the Constraint Satisfaction Probem (CSP) [4]. We wi ca this approach the CSP approach in the foowing. As concuded in Section IV D, we wi use the combined criterion (Equation 9) for the foowing comparisons. E.. Comparison with the COR Approach In [3], the mean probabiity that a moving object is detected at a ocation x in the FOV of a camera is caed an occurrence Utiity-based COR (e) (f) (g) (h) Camera Camera 2 Camera 3 Fig. 0. Experiment #2. Two camera handoffs by using the co-occurrence to occurrence ratio (COR) approach and the comparison with our approach. The first row are the resuts by our approach and the second row are the resuts by the COR approach. The camera seected to track the person provides a bue bounding box, otherwise it provides a yeow bounding box. Experiment #3: Comparison with the COR Approach (N C = 3, N P = 2, Indoor) In Fig., we show some error frames by using the COR approach. These resuts can be compared with Fig. 6 (Experiment #) where we use the same video for the

9 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 9 (a) (d) Camera Camera 2 Camera 3 Fig.. Experiment #3. Some camera handoff errors by the co-occurrence to occurrence ratio (COR) approach in a 3cameras, 2 persons case. The cameras that are seected for tracking a person provides a bue bounding box for that person, otherwise it provides green bounding box for the person in red and red bounding box for the person in green. proposed approach. By the comparison, we can notice that the COR approach can ony switch the camera to another one when the person is about to eave the FOV, but cannot seect the best camera based on other criteria. So, the number of handoffs by our approach is arger than that of the COR approach (See Tabe IV). If we use the definition of error as stated in Section IV C, the error rates for these two cases are compared in Tabe IV. Based on this error definition, the COR approach oses the fronta view of a person more easiy. Exampes are Fig. (b) (ose the person in red), (d) (ose the fronta view of the person in red) and (f) (ose the fronta view of the person in green). TABLE IV COMPARISON OF ERROR RATES FOR THE CO-OCCURRENCE TO OCCURRENCE RATIO (COR) APPROACH AND THE PROPOSED APPROACH Experiment #2 Experiment #3 # of handoffs Error rate # of handoffs Error rate COR % % Proposed % % E.2. Comparison with the CSP Approach The approach in [4] soves the camera assignment probem by using the constraint satisfaction approach. According to the key assumptions made in Section III B., we aow one camera to track mutipe persons but one person can ony be tracked by one camera. So, for each camera C j, we et a those persons that can be seen by this camera form a group g j. For instance, if, in our case, the camera C j can see person P and P 2, then the domain of g j, noted as Dom[g j ], is {{P }, {P 2 }, {P, P 2 }}. The constraint is set to be d i d j =, for i j, where d i b i is the camera assigned to track person P i, and b i and b j beong to Dom[g j ] and i j. By doing so, we mean that the persons to be tracked are assigned to different cameras. We changed some of the notations from [4] so that the notations in this section are not in confict with the notations used in the previous sections of this paper. Experiments for 3 cameras, 2 persons (Experiment #4) and 4 cameras, 4 persons (Experiment #5) cases are carried out under the above constraint to maximize the criterion 4 (Equation 9), using the BestSov agorithm in [4]. Experiment #4: Comparison with the CSP Approach (N C = 3, N P = 2, Indoor) (b) (e) (c) (f) Since our approach requires 5 iterations for the 3 cameras, 2 persons case (Experiment #3) to get acceptabe resuts, we aso use 5 backtracking steps in the CSP approach. Both the CSP approach and our proposed approach are abe to accommodate different criteria. Most of the time, the CSP approach can seect the best camera, based on our criterion and the error definition. So, we ony compare the number of handoffs and the error rates for this case in Tabe V. The resuts show that the CSP approach has higher error rates than our approach. TABLE V COMPARISON OF ERROR RATES FOR THE CONSTRAINT SATISFACTION PROBLEM (CSP) APPROACH AND THE PROPOSED APPROACH Experiment #4 Experiment #5 # of handoffs Error rate # of handoffs Error rate CSP % % Proposed % % Experiment #5: Comparison with the CSP Approach (N C = 4, N P = 4, Indoor) Since in this case, there are more persons and cameras invoved, we increase the number of backtracking steps and the number of iterations to 0. Because the performance of the CSP approach heaviy depends on the number of backtracks (the more backtracks it takes, the more accurate the resuts can be), as the number of cameras and persons goes up, the CSP approach wi miss the best camera with a high probabiity. Some of the errors for this case are shown in Fig. 2. There are errors when a person s fronta view is avaiabe but it is not chosen such as in Fig. 2 (b) (d) and (f), or when a person s fronta view is unavaiabe, the system chooses the camera with a smaer size person and farther from the center of the FOV, such as in Fig. 2(d) for the person in back. The high error rate for the CSP approach is due to its computationa cost. (a) (b) (c) (d) (e) (f) Camera Camera 2 Camera 3 Camera 4 Fig. 2. Experiment #5. Some error frames by using the Constraint Satisfaction Probem (CSP) approach for the 4 cameras, 4 persons case. The camera with bue bounding box for a person is seected to track the person. The cameras seected for tracking a person provides a bue bounding box for that person. Experiment #6: Further Comparison between the CSP and the Proposed Game Theoretic Approach Number of Iterations (N c = 3, N P = 0) Fig. 3 gives a comparison of number of iterations for our approach and the number of backtracks for the CSP approach for the case when the number of cameras is fixed to 3 and the number of persons goes up from to 0. We can see that

10 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK 0 athough the CSP approach can sove the camera assignment probem based on different user-defined tasks, it is computationay expensive as the compexity of the system increases. Fig. 3. Experiment #6. Comparison for the number of iteration or backtrack by the proposed utiity-based approach and by the Constraint Satisfaction Probem (CSP) approach. E.3. Comparison among Game Theoretic, COR and CSP Approaches Experiment #7: N C = 6, N P = 4, Outdoor In this section, we consider a more compicated case with 4 cameras and 6 persons. Because there are too many peope in the system, it wi be hard to observe if we mark the person in a the cameras that can see them. So, we ony draw the bounding boxes for those cameras which are assigned to track A A2 A3 A4 the specific person. Different coors are used to distinguish different persons. We ony dispay some typica resuts (Fig. 4) for each of the approaches that are compared. Because there are more cameras and persons invoved in this experiment than the previous ones, we increase the number of iterations to 20 for a the CSP and the proposed approach. For the proposed approach, we can notice that whenever there is a camera avaiabe to track a specific person, the camera assignment can be performed based on the pre-defined criteria. In Fig. 4 A6 (the Utiity-based approach group), we provide a case when the bargaining mechanism fais, i.e., the number of iterations is not arge enough to converge to the optima resut. In this figure, the person in red bounding box shoud be tracked by Camera based on an exhaustive cacuation which can be regarded as the ground-truth. The COR approach cannot decide which camera to seect based on the user suppied criteria. So most of the handoffs take pace when a person is eaving the FOV of one camera and entering the FOV of another camera. The CSP approach can dea with the suppied criteria to some extent, but since 20 backtracks are too few to reach the optima answer, the CSP oses the best camera easiy. The overa performance of these approaches is presented in Tabe VI. TABLE VI COMPARISON OF ERROR RATES FOR THE COR, CSP AND THE PROPOSED APPROACH Experiment #6 COR CSP Proposed Error rates 45.67% 2.96% 7.89% Utiity-based COR CSP A5 A6 A7 A8 B B2 B3 B4 B5 B6 B7 B8 C C2 C3 C4 C5 C6 C7 Camera C8 Camera 2 Camera 3 Camera 4 Fig. 4. Experiment #7. A comparison for the proposed utiity-based game theoretic approach, the COR approach and the CSP approach. Ony those cameras seected to track a person provide a bounding box for that person. V. CONCLUSIONS In this paper, we proposed a new principed approach based on game theory for the camera assignment and handoff probem. We deveoped a set of intuitive criteria in this paper and compared them with each other as we as the combination of them. Our experiments showed that the combined criterion is the best based on the error definition provided in Section I C. Since the utiities, input of the bargaining process, argey depend on the user-suppied criteria, our proposed approach can be task-oriented. Unike the conventiona approaches which perform camera handoffs ony when an object is eaving or entering the FOV, we can seect the best camera based on the pre-defined criteria. The key merit of the proposed approach is that we use a theoreticay sound game theory framework with bargaining mechanism for camera assignment in a video network so that we can obtain a stabe soution with a reasonaby sma number of iterations. The approach is independent of (a) the spatia and geometrica reationships among the cameras, and (b) the trajectories of the objects in the system. It is robust with respect to mutipe user-suppied criteria. The approach is fexibe since there is no requirement for a specific criterion that a user is obigated to use. A wide variety of experiments show that our approach is computationay more efficient and robust with respect to other existing approaches [3, 2]. We anayzed the infuence of a tracker on the proposed approach in Section IV B and compared our work with two other recent approaches both quaitativey and quantitativey. A the experiments used a physica camera network with rea

11 This artice has been accepted for pubication in a future issue of this journa, but has not been fuy edited. Content may change prior to fina pubication. Y.LI AND B.BHANU CAMERA ASSIGNMENT: A GAME-THEORETIC FRAMEWORK data in rea time. This incuded both indoor and outdoor environments with different numbers of cameras and persons. As compared to the other approaches, it is shown that the proposed approach has smaer error rates in a the experiments. The computationa efficiency of the proposed approach is aso verified quantitativey. This comparison shows that (a) COR approach cannot do any criteriondependent camera assignment. (b) As the number of cameras and persons in the system increases, the assignment ambiguity and faiure aso increase in the COR approach. (c) The CSP approach is task-dependent and can seect the best camera based on whatever criterion is provided by the user. (d) The CSP approach is computationay much more expensive than our approach. Our future work wi aow communication among cameras, which wi make the computationa framework and computationa resources decentraized and distributed. REFERENCES [] O. Javed, S. Khan, Z. Rasheed and M. Shah. Camera Hand-off: Tracking in Mutipe Uncaibrated Stationary Cameras. IEEE Workshop on Human Motion, [2] J. Park, P. C. Bhat, and A. C. Kak. A Look-up Tabe Based Approach for Soving the Camera Seection Probem in Large Camera Networks. Int Workshop on Distributed Smart Cameras, [3] Y. Jo and J. Han. A New Approach to Camera Hand-off without Camera Caibration for the Genera Scene with Non-panar ground. ACM Int Workshop on Video Surveiance and Sensor Networks, [4] F. Z. Qureshi and D. Terzopouos. Muti-Camera Contro through Constraint Satisfaction for Pedestrian Surveiance. AVSS [5] X. Zou and B. Bhanu. Anomaous Activity Detection and Cassification in a Distributed Camera Network. ICIP, [6] V. Kettnaker and R. Zabih. Bayesian Muti-camera Surveiance. CVPR, 999. [7] T. Chang and S. Gong. Bayesian Modaity Fusion for Tracking Mutipe Peope with a Muti-Camera System. European Workshop on Advanced Video-based Surveiance Systems, 200. [8] J. Kim amd D. Kim. Probabiistic Camera Hand-off for Visua Surveiance. ICDSC, [9] J. Kang, I. Cohen, and G. Medioni. Continuous Tracking within and across Camera Streams. CVPR, [0] O. Javed, K. Shafique, and M. Shah. Appearance Modeing for Tracking in Mutipe Non-overapping Cameras. CVPR, [] B. Song, A. Roy-Chowdhury. Stochastic Adaptive Tracking In A Camera Network. ICCV, [2] Q. Cai and J. K. Aggarwa. Tracking Human Motion in Structured Environments Using a Distributed Camera System. PAMI, 999. [3] G. Arsan, J. R. Marden and J. S. Shamma. Autonomous Vehice- Target Assignment: A Game-Theoretica Formuation. ASME Transactions on Dynamic Systems, Measurement, and Contro, specia issue on Anaysis and Contro of Muti-Agent Dynamic Systems, [4] L. Tessens et a. Principa View Determination for Camera Seection in Distributed Smart Camera Networks. ICDSC, [5] R. B. Myerson. Game theory-anaysis of Confict. Harvard University Press, Cambridge, MA, 99. [6] M. J. Osborne. An Introduction to Game Theory. Oxford University Press, [7] [8] Y. Li and B. Bhanu. Utiity-Based Dynamic Camera Assignment and Hand-off in a Video Network. ICDSC, [9] G. R. Bradski. Computer Vision Face Tracking for Use in a Perceptua User Interface. Inte Technoogy Journa Q2, 998. [20] [2] [2] M.J. Osborne and A. Rubinstein. A Course in Game Theory. The MIT Press, Cambridge, MA, 994. Yiming Li received the B.S. degree in eectrica Engineering from the South China University of Technoogy, Guangzhou, China, in 2006, the M.S. degree in eectrica engineering from the University of Forida, Gainesvie, FL, in She is currenty working toward the Ph.D. degree at the Center for Research in Inteigent Systems in the University of Caifornia, Riverside. Her research interests are in computer vision, pattern recognition and image processing. Her recent research has been concerned with camera networks. Bir Bhanu (S 72 M 82 SM 87 F 96) received the S.M. and E.E. degrees in eectrica engineering and computer science from the Massachusetts Institute of Technoogy, Cambridge, the Ph.D. degree in eectrica engineering from the Image Processing Institute, University of Southern Caifornia, Los Angees, and the MBA degree from the University of Caifornia, Irvine. Dr. Bhanu is the Distinguished Professor of Eectrica Engineering and serves as the Founding Director of the interdiscipinary Center for Research in Inteigent Systems at the University of Caifornia, Riverside (UCR). He was the founding Professor of eectrica engineering at UCR and served as its first chair (99-94). He has been the cooperative Professor of Computer Science and Engineering (since 99), Bioengineering (since 2006), Mechanica Engineering (since 2008) and the Director of Visuaization and Inteigent Systems Laboratory (since 99). Previousy, he was a Senior Honeywe Feow with Honeywe Inc., Minneapois, MN. He has been with the facuty of the Department of Computer Science, University of Utah, Sat Lake City, and with Ford Aerospace and Communications Corporation, Newport Beach, CA; INRIA- France; and IBM San Jose Research Laboratory, San Jose, CA. He has been the principa investigator of various programs for the Nationa Science Foundation, the Defense Advanced Research Projects Agency (DARPA), the Nationa Aeronautics and Space Administration, the Air Force Office of Scientific Research, the Office of Nava Research, the Army Research Office, and other agencies and industries in the areas of video networks, video understanding, video bioinformatics, earning and vision, image understanding, pattern recognition, target recognition, biometrics, autonomous navigation, image databases, and machine-vision appications. He is coauthor of the books Computationa Learning for Adaptive Computer Vision (Forthcoming), Human Recognition at a Distance in Video (Springer- Verag, 200), Human Ear Recognition by Computer (Springer-Verag, 2008), Evoutionary Synthesis of Pattern Recognition Systems (Springer-Verag, 2005), Computationa Agorithms for Fingerprint Recognition (Kuwer, 2004), Genetic Learning for Adaptive Image Segmentation (Kuwer, 994), and Quaitative Motion Understanding (Kuwer,992), and coeditor of the books on Computer Vision Beyond the Visibe Spectrum (Springer-Verag, 2004), Distributed Video Sensor Networks (Springer-Verag, 200), and Mutibiometrics for Human Identification (Cambridge University Press, 200). He is the hoder of U.S. and internationa patents. He has more than 350 reviewed technica pubications. He has been on the editoria board of various journas and has edited specia issues of severa IEEE TRANSACTIONS (PATTERN ANALYSIS AND MACHINE INTELLIGENCE; IMAGE PROCESSING; SYSTEMS, MAN AND CYBERNETICS PART B; ROBOTICS AND AUTOMATION; INFORMATION FORENSICS AND SECURITY) and other journas. He was the Genera Chair for the IEEE Conference on Computer Vision and Pattern Recognition, the IEEE Conference on Advanced Video and Signa-based Surveiance, the IEEE Workshops on Appications of Computer Vision, the IEEE Workshops on Learning in Computer Vision and Pattern Recognition; the Chair for the DARPA Image Understanding Workshop, the IEEE Workshops on Computer Vision Beyond the Visibe Spectrum and the IEEE Workshops on Muti-Moda Biometrics. He was the recipient of best conference papers and outstanding journa paper awards and the industria and university awards for research exceence, outstanding contributions, and team efforts. Dr. Bhanu is a Feow of the IEEE Computer Society, the American Association for the Advancement of Science, the Internationa Association of Pattern Recognition, and the Internationa Society for Optica Engineering.