Self-aware and Self-expressive Camera Networks

Transcription

1 1 Self-aware and Self-expressive Camera Networks Bernhard Rinner, Lukas Esterle, Jennifer Simonjan, Georg Nebehay, Roman Pflugfelder, Peter R. Lewis and Gustavo Fernández Domínguez Abstrat Reent advanes have enabled a dramati hange in amera networks. Smart ameras perform image analysis onboard, adapt their algorithms in response to hanges in their environments and ollaborate with other ameras in order to analyze the dynami behavior of objets in partly unknown environments. A fixed onfiguration is infeasible to manage the tradeoff among performane, flexibility, resoures and reliability in suh amera networks. We adopt the onepts of self-awareness and self-expression for autonomous monitoring of the state and progress of the amera network and adaptation of its behavior to hanging onditions. We desribe the building bloks for selfaware amera networks and demonstrate the key harateristis in simulation and a real amera network. Index Terms Self-awareness, self-expression, smart amera networks, real-time adaptability. I. INTRODUCTION Camera networks are now ubiquitous and have appliations in seurity, disaster response, environmental monitoring and smart environments, among others. Smart amera networks have emerged reently by bringing together advanes in omputer vision, embedded omputing, image sensors and networks. They are real-time, distributed, embedded systems that perform omputer vision tasks using multiple ameras [1]. In order to provide in-network proessing of aptured data, smart amera networks have to deal with various hallenges. Cameras analyze the highly dynami behavior of objets, are deployed in partly unknown environments and ooperate with neighboring ameras on demand. Future amera networks should be able to ahieve advaned levels of autonomous behavior to adapt themselves at runtime and learn behaviors appropriate to hanging onditions. A partiular hallenge is to manage the trade-off of onfliting objetives suh as high performane, low resoure onsumption and high reliability. A fixed onfiguration of the amera network is infeasible to overome these hallenges [2]. As a suessful alternative we adopt the onepts of selfawareness (SA) and self-expression (SE), translating them to omputational analogies and applying them to amera networks. Both onepts are new to the domains of omputing and networking. Self-awareness refers to the ability of a system to obtain and maintain knowledge about its state, behavior and progress, enabling self-expression, the generation of autonomous behavior based on suh self-awareness. Together, self-awareness and self-expression support effetive and autonomous adaptation of behavior to hanging onditions. Learning models online of the amera s state and ontext B. Rinner, L. Esterle, and J. Simonjan are with the Alpen-Adria-Universität Klagenfurt, Austria. G. Nebehay, R. Pflugfelder and G. Fernández Domínguez are with the Austrian Institute of Tehnology, Austria. P.R. Lewis is with Aston University, U.K. as well as deentralized deision making in the network are the fundamental tehniques for ahieving SA and SE. The building bloks and their interation are explained in order to build a omputationally self-aware and self-expressive amera network. Its features and apabilities are shown pratially with a distributed multi-amera traking appliation as an example. AUTONOMOUS MULTI-CAMERA COORDINATION Coordination is a fundamental problem in amera networks with the key objetive to dynamially assign ations to ameras in order to improve the performane of the network tasks. These tasks may inlude overage optimization, pan-tilt-zoom (PTZ) ontrol and traking and an be found in a myriad of different appliations suh as surveillane, transportation, seurity and ativity reognition [a, b]. The assigned ations determine the plaing and orientation of (mobile or PTZ) ameras, the sensing apabilities suh as resolution and frame rate, the data proessing and the data ommuniation. Researhers have adopted different approahes to oordinate multiple ative ameras suh as ontrol theory, game theory, state mahines, multi-agent systems, probabilisti approahes, and many other ad-ho approahes as well []. In entralized oordination the deision making is performed at a single node that reeives data from eah amera in the network. It introdues a signifiant omputation and ommuniation load but an ahieve a high task performane due to the availability of the entire network information in a single node. In deentralized oordination the deision making and the required data from the ameras are distributed among the network. In autonomous oordination eah amera individually deides what ation to take based on its loal assessment whih may inlude information from other ameras. The goal of multi-amera traking is to detet, loalize and trak moving objets suh as pedestrians or vehiles within the fields of views (FOV) of all ameras. Deiding whih amera is responsible for traking a speifi objet represents a typial oordination problem [d]. In entralized oordination the ameras send the traes of the objets within their FOV to a entral node whih then selets the best trae. In autonomous oordination eah amera deides by its own when and to whom to handover the traking responsibility. Deriving the handover deision based on inomplete information and with amera s limited resoures is a fundamental hallenge. SA and SE

2 2 helps to overome some of the diffiulties of oordinating the traking responsibility and is able to ahieve robust, flexible and salable multi-amera traking ontrol with low omputation and ommuniation overhead. Referenes [a] [b] R. J. Radke, A Survey of Distributed Computer Vision Algorithms. in Handbook of Ambient Intelligene and Smart Environments. Springer, 2009, pp B. Bhanu, C. V. Ravishankar, A. Roy-Chowdhury, H. Aghajan, and D. Terzopoulos, Eds., Distributed Video Sensor Networks. Springer, [] S. Chen, Y. Li, and N. M. Kwok, Ative Vision in Roboti systems: A survey of reent developments, The International Journal of Robotis Researh, vol. 30, no. 11, pp , [d] M. Taj and A. Cavallaro, Distributed and deentralized multiamera traking, IEEE Signal Proessing Magazine, vol. 28, no. 3, pp , May II. SELF-AWARENESS AND SELF-EXPRESSION In ommon with many other emerging omputational systems, smart amera networks fae the hallenge of operating in a omplex interonneted world, where they interat with people and eah other in ways whih are diffiult to understand and predit. In order to meet this hallenge, suh systems must possess an inreased level of awareness, both of the world around them and of themselves. But we must not only attempt to repliate the self-awareness apabilities of humans in omputers; there will be important differenes. Instead, the notion of omputational self-awareness is being developed [3], inspired by onepts of human self-awareness. Conepts of self-awareness need to be translated from fields suh as psyhology and applied to the very different domain of omputing. Systems possessing suh omputational selfawareness will be able to ontinuously learn and adapt during their lifetime. They will build up an awareness of themselves and of their own experiene of the world whih they inhabit. And they will also need to be aware of the way in whih they themselves are aware of these things. The study of self-awareness emerged as a field within psyhology in the 1960 s. Morin [4] defines self-awareness as the apaity to beome the objet of one s own attention. A prerequisite for this is the apability to monitor or observe oneself. Our work is onerned with taking inspiration from self-awareness theories in order to better engineer omputational systems. We therefore aim at bridging the gap between psyhology [5] and engineering (e.g., [3]). As part of this, we developed an engineering methodology for self-aware and selfexpressive omputing systems, inluding a general referene arhitetural framework, able to be adapted to a wide range of appliations. In Setion III, we desribe our work to apply this methodology, in partiular onretizing the referene arhitetural framework for the autonomous oordination of nodes in a multi-amera network. This framework embodies various onepts from selfawareness theory. One important one is that self-awareness an be an emergent property of olletive systems, even when there is no single omponent with a global awareness of the whole system [6]. This is a key observation whih an ontribute to User Input Images Objetives & Constraints Self-Aware and Self-Exrpessive Camera Node Objet Traking Self-Awareness Resoure Monitoring Models Self-Expression Objet Handover Strategy Seletion Topology Learning Network Reeived Bids Sent Bids Autions Fig. 1: The arhiteture of a self-aware and self-expressive amera node omposed by six building bloks. the design of self-aware systems: one need not require that a self-aware system has a node with global knowledge. Also in this framework, the term self-expression is used to refer to behavior based on self-awareness, and an also be onsidered both at the node level [3] and olletive level [7]. There are several lusters of researh in omputer siene and engineering whih have used the term self-awareness expliitly [5]. However, our work represents the first holisti framework for desribing and benhmarking the selfawareness properties of omputational systems, and the benefits that omputational self-awareness an bring. While we use a speifi appliation ontext, as has been done in related work [8], [9], our work also generalizes this idea. Indeed, this artile demonstrates the notion of omputational selfawareness in our work on smart amera networks. III. APPLICATION DESIGN WITH SA&SE BUILDING BLOCKS We apply our generi arhitetural patterns [3] to develop a flexible SA&SE multi-amera appliation. The key approah here is to instantiate dediated software omponents and their interations. These omponents serve as distint building bloks for the onrete multi-amera traking appliation in our ase. Fig. 1 depits the overall arhiteture of a self-aware and self-expressive amera node ontributing to the multiamera traking appliation. Eah building blok has speifi objetives and interats with other bloks in the network. Selfawareness is realized by individual bloks for Objet Traking, Resoure Monitoring, and Topology Learning. Instead of relying on predefined knowledge and rules, these bloks utilize online learning and maintain models for the amera s state and ontext. These models serve as input to self-expression whih is omposed of the Objet Handover and the Strategy Seletion bloks. Finally, the Objetives & Constraints blok represents the amera s objetives and resoure onstraints, where both have a strong influene on the other bloks. We an ompose a truly deentralized, self-aware and self-expressive amera network by aggregating our amera nodes. The building bloks of eah node an be implemented by diverse algorithms from omputer vision, online learning and deision making. However, resoure awareness was one guiding priniple for our design. Thus, all building bloks

3 3 must be able to exeute in real-time on our resoure limited embedded amera platforms. A. Objet Traking This building blok employs a simple appearane-based approah for visual objet traking. The hosen method fulfills the ameras omputational resoure and real-time requirements and ahieves for the given test environment an aeptable level of robustness against dropped frames, olusions and disappearane of objets. In a first step, we identify foreground pixels in eah amera by omparing the amera image to a bakground image that eah amera learns individually about its field of view. This approah assumes in its simplest form stati ameras. Foreground onstitutes moving objets that omprises objets of interest that should be traked. These foreground pixels are then grouped into andidate objets based on their onnetedness. In a seond step, we perform an assoiation of these andidate objets to a template database that ontains the objets of interest. To this end, we employ a measure of similarity aording to [10] that we also interpret as the onfidene in the validity of the assoiation. It is important to note that the approah does not aim to ompete with the state of the art in visual multi-amera objet traking targeting general environmental onditions. Instead, the method serves as an exemplary implementation of a selfaware objet traking building blok, suffiient to validate the SA&SE framework. B. Objet Handover To oordinate the objet traking responsibilities in the amera network, we apply a novel market-based handover approah [11]. Here, the ameras treat objet traking responsibilities as goods, providing some utility over time. The ameras an deide in a self-expressive manner on their own when to sell traking responsibilities to other ameras using virtual autions. Whenever a amera deides to sell an objet, it initiates an aution for this partiular objet by transferring an objet desription to the other ameras. The reeiving ameras searh their own FOV for the objet and value the objet based on detetion onfidene and visibility. These ameras return their valuation as a bid. The autioneering amera selets the highest bidder and transfers the traking responsibility. We use the Vikrey aution mehanism, whih sells the good to the highest bidder for the seond highest prie, to make truthful bidding the dominant strategy among the partiipating ameras. Fig. 2 illustrates the key steps of the market-based handover whih is a fully deentralized mehanism relying only on autonomous deisions of ameras. An important question for the selling amera is to whom to send the aution invitations. Without any a priori knowledge about the network topology, invitations ould be broadasted to all ameras. By following this strategy the best amera for taking over the traking responsibility will reeive an invitation (and may respond with the highest bid). However, the broadast strategy auses a signifiant ommuniation and omputation load, i.e., beause eah amera has to perform an objet detetion after reeiving the invitation. C. Topology Learning If the autioneering ameras are aware of the potentially best ameras in their neighborhood, this knowledge an be exploited to signifiantly redue the overhead. Suh topologial information an be initially assigned to the ameras or omputed by means of multi-amera alibration during the deployment of the amera network. However to improve selfawareness, we learn the topology by observing the bidding behavior of ameras over time. Eah amera individually keeps trak of their loal neighbors and uses artifiial pheromones to express the likelihood of a handover to that amera. Whenever a handover has taken plae, the artifiial pheromone to the sueeding amera is strengthened. If no trading ours, pheromones evaporate over time. This mehanism enables eah amera to deal with network unertainties and to adapt to hanges in their neighborhood topology, e.g., aused by adding, removing or failure of ameras or hanges in the movement pattern of the objets. We exploit the learned neighborhood topology by three different ommuniation strategies for the handover: broadast autions to all ameras (BROADCAST), a smooth probabilisti multiast (SMOOTH) and a threshold-based probabilisti multiast (STEP) [11]. The SMOOTH strategy sends aution invitations to all neighbors with probability normalized to the urrent pheromone level. The STEP strategy sends invitations to all neighbors with pheromone level above a ertain threshold and to neighbors below the threshold with some (low) probability. We further distinguish whether to send out invitations at regular intervals (ACTIVE) or only when the objet is about to leave the FOV (PASSIVE). We then end up with six different self-expressive handover strategies by ombining the approah on whom to invite and when to send out the invitations. Obviously, the seleted strategy influenes the ahieved traking utility as well as ommuniation and omputational overhead. D. Strategy Seletion While the six handover strategies allow to trade off ommuniation overhead against traking utility and hene influene the behavior of the network, seleting a strategy is a diffiult deision. The performane of eah strategy strongly depends on fators suh as the plaement of the ameras, the movement patters of the objets and the objet traking algorithm. In priniple, we an follow three approahes for strategy seletion: (i) a homogeneous seletion, where all ameras selet the same strategy at deployment time, (ii) a heterogeneous seletion, where the ameras an selet their strategy individually at deployment time, and (iii) a dynami seletion, where eah amera an selet its strategy during runtime. We use online learning algorithms, speifially multi-armed bandit problem solvers within eah amera to learn the appropriate strategy for eah node during runtime. Bandit solvers balane exploitation behavior, where a amera ahieves high performane by using its urrently known best strategy, with exploration, where the amera explores the effet of using other strategies to build up its knowledge [12]. We use standard bandit-solvers from the literature, namely Softmax,

4 4 Aution Invitation Bids (a) (b) Traking Responsibility () Fig. 2: Illustration of the market-based handover approah: An objet of interest is traked by amera 2 indiated by the green FOV (a). 2 initiates an aution (blak arrows) for the objet when it is about to leave the FOV (b). Cameras 3 and 4 detet the objet in their FOVs (orange) and return bids (blak arrows) based on their objet valuation to 2 (). 3 has offered the highest bid and wins the aution (d). The traking responsibility is transferred from 2 to 3 (blak arrow) and an artifiial pheromone is added (indiated as red line between 2 and 3 ). (d) Epsilon-Greedy and UCB1. Dynami strategy seletion leads to another level of self-aware and self-expressive behavior of the amera network and is able to ahieve a more Pareto effiient global performane than with any stati seletion. E. Resoure Monitoring Resoure monitoring is an important aspet of omputational self-awareness, and its main objetive is to observe the available resoures on the amera nodes. The monitored data is further used to build up models of resoure onsumption for eah task a amera is apable of performing. This information allows a self-expressive blok (e.g., strategy seletion) to reason not only about the performane of eah task but also about its respetive resoure onsumption. In our network, we urrently monitor required proessing power, available and alloated memory, and network traffi. F. Constraints and Objetives Eah amera has some onstraints and objetives whih need to be onsidered for self-aware and self-expressive operation. In our system onstraints speify some limitations of the available resoures (proessing, memory and networking) and help to deide on whether to bid for an objet. On the other hand, objetives are related to the behavior of the ameras and speify, for example, some quality of servie parameters or speifi tasks the ameras should ahieve. IV. SMART CAMERA NETWORK A. Experimental Study: Camera Network Setup Fig. 3 depits the setup of the smart amera network used for our experimental study. Four ameras (1 4) are mounted in a laboratory room with overlapping FOVs. Cameras 5 and 6 are plaed in the orridor and lounge area, respetively. Our heterogeneous network is omposed by different hardware platforms. Cameras 1 to 4 are equipped with Atom proessors and onneted via wired Ethernet. Cameras 5 and 6 are based on Pandaboards equipped with ARM proessors and use WiFi for ommuniation. All ameras run standard Linux and a distributed publish-subsribe middleware system [13] to provide a flexible software platform for software development. The building bloks have been implemented in C++ and C# using the middleware servies for ommuniation and ontrol. B. Traking Results For the evaluation of our objet traking blok we use the smart amera network with various people walking through the indoor environment (as depited in Fig. 3). We apply state-of-the-art metris (e.g., [14]) for the evaluation. Fig. 4(a) summarizes the results of a typial senario of onurrently traking three seleted persons independently walking around in the indoor environment for around 120 seonds. During this senario, the persons were not ontinuously visible to all ameras; at some points, partiipants were not seen by any amera at all. The ameras mounted in the laboratory ahieved

5 5 Laboratory Corridor Lounge Fig. 3: The illustration above shows the smart amera network omposed of six ameras deployed in an indoor environment. The ameras are depited by a blak amera symbol and their FOVs are indiated by orange lines. Snapshots of six ameras are indiated by blue lines. Those images show the status of objet traking over time; three people are traked by the system marked by red, blue and green bounding boxes. better detetion and traking results. The performane of ameras 5 and 6 degraded slightly due to moderate hanges in lightning and objet appearane. Overall, the experimental results show suffiient performane of the objet traker for orret objet handover in the given test environment. C. Topology Learning Through our self-aware topology learning blok eah individual amera builds up a loal neighborhood relationship graph. Fig. 4(b) shows the aggregated graph for the entire network after a test run in our smart amera network. The thikness of the red lines indiate the strength of the artifiial pheromone deposit on this link whih orresponds to the probability of an objet transiting between the onneted ameras. Initially, links are reated between the amera in the lounge (amera 6) and those in the laboratory (ameras 1-4) due to misdetetions of amera 5. Through the evaporation of the artifiial pheromones, ameras an not only deal with errors indued by the traking blok but also with hanges in the topology due to hardware errors or vandalism. Over time these links evaporate and a qualitatively orret neighborhood graph emerges. D. Communiation and Utility Trade-off We evaluated the effet of the handover strategy on the overall traking utility and ommuniation overhead. Figure 5(a) depits the trade-off of utility and ommuniation for homogeneous strategy seletion in our smart amera network. The utility is defined as the aggregated traking utility of all ameras, and the ommuniation is defined as the number of all sent aution messages during the entire traking operation. Both utility and ommuniation values are normalized by those from the ACTIVE BROADCAST strategy. The six strategies result in six different trade-offs for utility and ommuniation. Figure 5(b) ompares the ahieved trade-off for homogeneous, heterogeneous and dynami strategy seletion in our CamSim1 simulation tool [12]. Obviously, heterogeneous seletion (blak rosses) leads to many more outomes in the objetive spae. The extension of the Pareto effiient frontier brought about by heterogeneity is also apparent. However, it is also lear that the outomes of many heterogeneous strategies are dominated, and many are stritly worse than the original outomes from the homogeneous strategies. We an learly see the benefit of self-expressive behavior. Dynami strategy seletion, here implemented using reinforement learning (olored symbols), is able to outperform the stati (homogeneous und heterogeneous) strategies and to extend the Pareto front. V. C ONCLUSION As demonstrated in this paper, self-awareness and selfexpression are fundamental onepts for developing amera networks apable of learning and maintaining their topology, distributedly performing objet detetion and traking handover as well as autonomously seleting strategies to ahieve 1

6 6 Camera number Objet detetion: Sensitivity Objet traking: CDT (a) 6 (b) Fig. 4: Results for objet traking and topology learning for onurrently traking three seleted persons for around 120 seonds. The table (a) shows the sensitivity of objet detetion and the orret deteted traks (CDT) for eah amera. The graph (b) shows the learned topology after 60 seonds by exploiting the trading behavior. The thikness of the red line indiates the pheromone level of the link Ative SMOOTH Ative STEP Ative Broadast Ative SMOOTH Ative STEP Ative Broadast 0.93 Utility 0.9 Passive Broadast Utility Passive Broadast 0.85 Passive SMOOTH Passive STEP Homogenous Passive SMOOTH Passive STEP Heterogeneous Homogeneous Softmax ((Temperature 0.1) Softmax ((Temperature 0.2) Epsilon Greedy (Epsilon = 0.1) UCB Communiation (a) Communiation Fig. 5: Performane for two exemplary senarios from our smart amera network (a) and our simulation environment (b) showing homogeneous (red and yellow squares), heterogeneous (blak rosses), and dynamially learned strategy assignments (olored symbols, representing different reinforement learning strategies evaluated). The results have been normalized by the maximum value of the ACTIVE BROADCAST strategy and are averages over 30 runs with 1000 time steps eah [12]. (b) more Pareto effiient outomes. The entire proessing is enapsulated into six building bloks embedded onto resourelimited smart amera nodes and aggregated into a ompletely deentralized and thus salable network. However, omputational self-awareness and self-expression is not limited to amera networks. In fat, we are onfident that SA and SE ould serve as an enabling tehnology for future systems and networks meeting a multitude of requirements with respet to funtionality, flexibility, performane, resoure usage, osts, reliability and safety. ACKNOWLEDGMENT This work has been partially supported by the EU FP7 program under grant agreement no REFERENCES [1] M. Reisslein, B. Rinner, and A. Roy-Chowdhury, Smart Camera Networks, Computer, vol. 47, no. 5, pp , May [2] J. C. SanMiguel, K. Shoop, A. Cavallaro, C. Miheloni, and G. L. Foresti, Self-Reonfigurable Smart Camera Networks, Computer, vol. 47, no. 5, pp , May [3] F. Faniyi, P. R. Lewis, R. Bahsoon, and X. Yao, Arhiteting self-aware software systems, in Pro. Working IEEE/IFIP Conferene on Software Arhiteture (WICSA) IEEE, 2014, pp [4] A. Morin, Levels of onsiousness and self-awareness : A omparison and integration of various neuroognitive views, Cons. and Cog., vol. 15, no. 2, pp , [5] P. R. Lewis, A. Chandra, S. Parsons, E. Robinson, K. Glette, R. Bahsoon, J. Torresen, and X. Yao, A survey of self-awareness and its appliation in omputing systems, in Pro. Fifth IEEE International Conferene on Self-Adaptive and Self-Organizing Systems Workshops (SASOW). IEEE Computer Soiety Press, 2011, pp [6] M. Mithell, Self-awareness and ontrol in deentralized systems, in Working Papers of the AAAI 2005 Spring Symposium on Metaognition in Computation. AAAI Press, 2005, pp [7] F. Zambonelli, N. Biohi, G. Cabri, L. Leonardi, and M. Puviani, On self-adaptation, self-expression, and self-awareness in autonomi servie omponent ensembles, in Fifth IEEE Conferene on Self- Adaptive and Self-Organizing Systems, SASOW 2011, Ann Arbor, MI, USA, Otober 3-7, 2011, Workshops Proeedings, 2011, pp [Online]. Available:

7 [8] M. Cox, Metaognition in omputation: A seleted researh review, Artfiial Intelligene, vol. 169, no. 2, pp , [9] A. Agarwal, J. Miller, J. Eastep, D. Wentziaff, and H. Kasture, Selfaware omputing, MIT, Teh. Rep. AFRL-RI-RS-TR , [10] D. Comaniiu, V. Ramesh, and P. Meer, Kernel-based objet traking, IEEE Transations on Pattern Analysis and Mahine Intelligene, vol. 25, no. 5, [11] L. Esterle, P. R. Lewis, X. Yao, and B. Rinner, Soio-eonomi vision graph generation and handover in distributed smart amera networks, ACM Transations on Sensor Networks, vol. 10, no. 2, pp. 20:1 20:24, [12] P. R. Lewis, L. Esterle, A. Chandra, B. Rinner, J. Torresen, and X. Yao, Stati, dynami and adaptive heterogeneity in distributed smart amera networks, ACM Transations on Adaptive and Autonomous Systems, p. 30, 2015, in print. [13] B. Dieber, J. Simonjan, L. Esterle, G. Nebehay, R. Pflugfelder, G. Fernandez, and B. Rinner, Ella: Middleware for Multi-amera Surveillane in Heterogeneous Visual Sensor Networks, in Proeedings of the Seventh ACM/IEEE International Conferene on Distributed Smart Cameras, Palm Springs, USA, [14] A. Baumann, M. Boltz, J. Ebling, M. Koenig, H. S. Loos, M. Merkel, W. Niem, J. K. Warzelhan, and J. Yu, A review and omparison of measures for automati video surveillane systems, EURASIP Journal on Image and Video Proessing, vol. 2008, no. 4,