Image-based Automatic Object Localisation in ispace Environment

Transcription

1 Magyar Kutatók 10. Nemzetközi Szimpóziuma 10 th International Symposium of Hungarian Researhers on Computational Intelligene an Informatis Image-base Automati Objet Loalisation in ispae Environment Péter Zanaty 1, Péter Koroni 2,3, Gábor Sziebig 4, László A. Jeni 5 1 Narvik University College, Department of Sientifi Computing, Eletrial Engineering an Spae tehnology, peter.zanaty@gmail.om 2 Buapest University of Tehnology an Eonomis, Department of Mehatronis, Optis an Applie Informatis 3 Computer an Automation Researh Institute, koroni@sztaki.hu 4 Narvik University College, Department of Inustrial Engineering, gabszi@hin.no 5 The University of Tokyo, Institute of Inustrial Siene, laszlo@hlab.iis.utokyo.a.jp Abstrat: Loalization is a funamental task in mobile robotis an in inoor environments we an use various sensors to solve this problem. In the Intelligent Spae environment we an use laser range finers or ultrasoni positioning systems to loalize an trak mobile robots. Nevertheless, our final goal is to substitute these sensors an aomplish this task using just ameras. In this paper, we show the feasibility of etermining the robot's loation base on the images reeive from a single amera. In our experimental room we use a surveillane amera, whih an be ontrolle to pan/tilt in orer to hange the point of view. The amera ha been mounte on the eiling an six preset positions were selete to over the whole area. The objet reognition is base on olour spae filtering an ontour etetion. Finally, the ontours of the etete objets are transforme from the image spae to the worl oorinate system an the polygons are reue to simpler ones. The system is able to etet not only the position of the objets, but their orientations. Keywors: ispae, image proessing, loalisation, alibration 1 Introution 1.1 Intelligent Spae The Intelligent Spae (ispae) is a onept of ombining eletrial evies together as istribute intelligent networke evies (DIND) for the purpose of supporting people in it [1,2]. The DINDs are proessing the obtaine information 501

2 P. Zanaty et al. Image-base Automati Objet Loalisation in ispae Environment in orer to unerstan the urrent state of the spae they are in. This information forms the basis of governing the ifferent atuators in it (see Figure 1). Figure 1 Intelligent Spae Suh intelligent environments are able to wath what is happening in them, buil a moel of them, ommuniate with their inhabitants an at base on the eisions they make. Espeially the apability of the environment to at as a ontextsensitive user interfae (e.g. to respon to gestures), an the reation in ertain situations (e.g. aients, intruers) promises a range of possible appliation senarios suh as intelligent hospital rooms, fatory, asylum for the age, et. There is a number of similar researh approahes to ispae, a few of them are the following: EasyLiving projet [3] at Mirosoft Researh is onerne with the evelopment of arhiteture an tehnologies for intelligent environments whih allow the ynami aggregation of iverse I/O evies into a single oherent user experiene. Projet Oxygen at MIT aims to reate a pervasive, human-entere omputational environment [4]. The Interative Workspaes Projet at Stanfor University has a basi motive is to explore new possibilities for people working together in tehnology-rih spaes [5]. Living with Robots an Interative Companions projet is a ollaboration of 10 European partners speialize in psyhology, ethology, humanomputer interation, human-robot interation, robotis an graphial haraters [6]. These projets are more or less the same initiatives targeting a ifferent auiene therefore having ifferent portfolio. EasyLiving fouses on extening the urrent esktops to the homes by expaning the user experiene starting from the PC. 502

3 Magyar Kutatók 10. Nemzetközi Szimpóziuma 10 th International Symposium of Hungarian Researhers on Computational Intelligene an Informatis Projet Oxygen applies hanhel an embee evies to provie user entri servies. The interative workspaes projet fouses on augmenting a eiate meeting spae with large isplays, wireless or multimoal evies, an seamless mobile appliane integration. While the LIREC projet aims to establish a multifaete theory of artifiial long-term ompanions relying on results of soial sienes suh as etymology. On the other han the onept of ispae is more robotis an inustrial tehnology inspire. There is a onsierable effort of researh on sensor networks, mobile robot ontrol, an Ubiquitous Computing [7], whih also is more or less relate to the ispae initiative. Often mobile robots are introue into an intelligent spae as atuators. Traking humans an traking an ontrolling robots in ispae are omplex tasks, for a review of urrent researh on this topi relate to ispae see [8]. In this paper yet another possible extension onto the onept is isusse: remote ontrol an interation with an ispae. Sensory ata may nee to be transmitte to a remote site for proessing an ontrol ata may nee to be sent bak to agents in an ispae. Moreover, remote ontrol by humans of a istant ispae may be neessary an optimal. Possible appliation senarios of telemanipulation inlue: operating in omains whih are not possible by simple means (e.g. miro an nanomanipulation) hanling tasks in hazarous environments (e.g. subsurfae, spae or nulear plant) speialize work on istant loations (e.g. brain surgery) 1.1 Narvik ispae Room At Narvik University College a temporary ispae onept room was establishe, for the purpose of ispae relate experiments (see Figure 2). Figure 2 Narvik ispae room 503

4 P. Zanaty et al. Image-base Automati Objet Loalisation in ispae Environment The room originally is esigne for testing heating equipments an measuring the gas by prouts in various ventilation onitions. To be able to hanle the interation with the reality, the loalization of objets is neessary. Therefore, it was essential to install lightweight, easy to setup - easy to remove, extension to the room to meet the nees of reating a temporary test room for intelligent spae. 2 Objet Reognition via Surveillane Camera This setion eals with the etails of objet reognition at Narvik ispae onept room. The best solution for equipping the room with objet reognition faility was to use web amera. The room is approximately 4.8 meters long an 3.6 meters eep with the height of 2.5 meters, this means that at least 6 normal (60 egree fiel of view) ameras are neee to over the whole room. As six ameras were not available, the only solution was to use a surveillane amera, whih an be ommane to pan/tilt in orer to hange the point of view. The atual amera moel is the WVC200 from Linksys whih is esigne for home or small ompanies to survey their properties while they are not present. Using the libcurl library a amera ontroller has been reate, whih an ontrol the amera via http requests. The ontroller also requests the live MJPEG stream from the amera. The MJPEG heaer is foun out to ontain the following useful information: total size of the frame, with an height of the frame, offset of the hunk, size of the hunk ata an a timestamp. Aoring to that a stream apturer is onstrute, whih, using the FreeImage library, parses the stream from the amera an transforms it to raw bitmap frames. 2.1 Moel of the Camera The amera ha been mounte on the eiling of the room in the entral part, an six presets were selete to over the whole area of the room. These presets an be reprogramme automatially using the reate amera hanler. The area visible by the ameras in this setup is visible at Figure

5 Magyar Kutatók 10. Nemzetközi Szimpóziuma 10 th International Symposium of Hungarian Researhers on Computational Intelligene an Informatis Figure 3 Six amera presets overing the whole area of the room To use the raw ata from the amera the mapping between the atual pixels an their position in the real spae ha to be one. There are numerous papers ealing with amera alibration, one popular metho is to use Tsai's amera alibration. The amera is alibrate aoring to a relaxe version of the propose metho in [9]. The following parameters represent a given state of the amera: amera loation at Eye Eye, Eye, Eye ) ( x y z entral pixel of the amera is mappe to At At, At, At ) ( x y z the iretion whih is onsiere up vetor by the amera is Up Up, Up, Up ) ( x y z resolution of the amera is 640x 480 whih is onstant respetive foal length for x an y axis f x an enter of the istortion x, ) ( y the raial amera istortion oeffiient 1 k The metho whih transforms a point from the real spae to the ameras piture an be ivie into two steps, first is the extrinsi transformation whih transforms the global oorinates to amera oorinates. Then the intrinsi transformation transforms the amera oorinates to atual pixels. f y 1 Experiments have shown that the tangential istortion is negligible, an using only the raial istortion results in a more leaner moel in terms of implementation. 505

6 P. Zanaty et al. Image-base Automati Objet Loalisation in ispae Environment The extrinsi transformation (projetion) is the following: x y 0 ( x ) = 0, y,, where ( x, y ) ( ) T 0, z0 x, y, z R z0 z0 0 = (1) an T S T R = U F T, where F Up At Eye S F S =, F =, U =, F Up At Eye S F Up Up = (2) Up where x, y are amera oorinates an x, y, z are the global oorinates. As we an see this transformation is not invertible, but if we know the istane of the point to the amera, or for instane the objet is loate on the floor, then we an also make the inverse transformation. The intrinsi transformation (raial lens istortion) is alulate the following way: x i = x + x fx an i y y y = + y f (3) 2 where ( x, y ) = ( x, y )(1 + k r ) an r = x + y (4) an x i, yi are atual pixels an y x, are from (1). To invert this transformation we nee to be able to alulate the original raius 2 2 from x, y ). Lukily the square of raius of these oorinates ( r = x + y ) ( 2 2 an the square of raius of the original oorinates ( r = x + y ) satisfies the following equation: 2 3 r = r (1 + k r ) => r = k r + r (5) This equation shoul have only one solution whih is less than one. Normally it will have three solutions: one whih is negative, one whih is larger than one, an one whih will be our the solution. This is eutive from the root properties of this speifi type of ubi equations. This means that solving this equation with the use of Carano's formula results in being able to invert this transformation. In orer to alibrate the amera a gri with the istane of 40 entimetres between ajaent points has been reore from the ifferent presets. Knowing both the real positions of the gri points an the image positions, a Matlab sript was reate as to fin out the ieal parameters for the moel esribe above. The benefit of using this moel rather than normal projetion is visualize for one preset with Figure

7 Magyar Kutatók 10. Nemzetközi Szimpóziuma 10 th International Symposium of Hungarian Researhers on Computational Intelligene an Informatis Figure 4 The lens istortion amera moel To alulate the parameters a Matlab program has been reate. It tries to minimize the error of the moel, an an fin aequate parameter results. 2.2 Automate Position an Orientation Reognition The amera hanler is apable to instrut the amera to go to a given preset, the response time for these ommans is a varying, so etetion whether the amera has reahe the position is vital. Consiering some physial properties, it an be assume that the amera annot hange its loation in less time then T 1. Similarly, we an assume that if it was operating orretly, then the worst ase response time for the amera to reah its position woul not exee T 2. This means that the amera shoul etet the image getting stable in [ T 1, T2 ] time-frame. Deteting if the image is stable an be one with alulating average absolute ifferene between onseutive frames using the OpenCV library. Objet reognition from visual ata is harer than one might notie. Human visual proessing is one of the most avane systems known in the universe [10]. Pretening that we an automate the ientifiation of objets easily using a amera an a omputer is pointless. Instea we an make a rational onstraint to eal only with ertain objets. To able to ifferentiate these objets from eah other an their surrounings, labels with ifferent olours are use. To provie a faility with whih labelle 507

8 P. Zanaty et al. Image-base Automati Objet Loalisation in ispae Environment objets an be reognize the OpenCV library is use. The introue objet reognition proess is the following: filter the image in olour spae (what is in the same olour as the label) etet the ontours (where are possible objets) transform the ontours into real oorinates (remove amera istortion) filter the ontours (oes the ontour looks like what we expet) smooth the ontour (if we want to etet orientation) The OpenCV library provies routines to filter an image in HSV 2 olour-spae then to etet the ontours in it. In orer to separate the noise whih oul be reognize from the real objets, these ontours shoul be filtere. First the area of the ontour is alulate then a bouning box is aligne to the ontour. Using these it is possible to filter out most of the noise an only get the ontours of esire objets. These ontours are transforme from image oorinates to real oorinates, an then the polygons are reue to simpler ones (typially from aroun 150 vertex to 3 or 4 vertex). To be able to etet not only the position of these objets but also the orientation a speial isoseles triangle shape is use, whih has a onsierably smaller sie an two larger sies. To esribe an objet (label) the following ata is neee (see Table 1): Property Type Desription rangeno integer Number of olour filter ranges range[] CvSalar[2] HSV range enpoints of the olour filter area float[2] the minimal an maximal area of the objet in sieratio float[2] ratio of the sies of the objet 2 m with float[2] minimal an maximal with of a bouning retangle height float[2] minimal an maximal height of a bouning retangle objheight float estimate height of the objet from the floor x float[2] min. an max. x oorinate of the bouning retangle y float[2] min. an max. y oorinate of the bouning retangle Table 1 Properties esribing reognizable objets 2 HSV (Hue Saturation Value) is a olor-spae whih esribes the pereptual olor relationships; using HSV in olor filtering results in more intuitive bounaries 508

9 Magyar Kutatók 10. Nemzetközi Szimpóziuma 10 th International Symposium of Hungarian Researhers on Computational Intelligene an Informatis These results ha been ombine into an automate amera system, whih an hange its position between the presets automatially an seek the whole room for reognizable objets. If any objet it etete, the ispae framework will be notifie through the preefine interfaes along with the position an orientation ata. 3 Camera System Test The amera system is esribe in Setion 2 an the applie moel is isusse in etails in Setion 2.1. For an illustration of the test room see Figure 5. Figure 5 The test room in Narvik, Norway To assist an verify the objet reognition moule, a graphial interfae has been reate. The graphial interfae serves multiple purposes. One of the purposes is to test the ommuniation between the moule an the amera: the piture oming from the amera an be viewe iretly by using this GUI. The seon option is to omman an alibrate the presets of the amera via this tool. Using these ommans one an navigate the amera with the amera tester moule, an see if it is working orretly. Also it is possible to reor the presets automatially, if the amera oes not know them yet or for some other reason the presets have beame orrupte. Thir purpose of this GUI is to help amera alibration. When the amera moel parameters are not yet ajuste to a room or for some reason these properties have 509

10 P. Zanaty et al. Image-base Automati Objet Loalisation in ispae Environment hange (e.g. the position or the iretion of the amera has been slightly hange), alibration is essential. Calibration of the amera an be broken own to three steps. First we nee to plae istintive markers on the floor (or anywhere in the room what is visible by the amera at the given preset) an measure the absolute positions of these markers. Seon step is to get the pixel oorinates of these markers from the piture reore by the amera at the given preset. Thir step is to alulate the amera parameters. The GUI itself provies a faility to reor these pixel-oorinates from a given preset, by magnifying the piture uner the mouse ursor, an reoring the oorinates of the mouse liks. The fourth funtion of the GUI is to ai the objet reognition proess by gathering olour information from the amera. There are two funtions to ease the reation of HSV ranges. The first is when the user liks with the right button on the piture the appliation prints out the olour of the pixel an isplays a winow of the olour to the user. The seon feature is to be able to speify a polygon on the piture whih will be the region of interest, an then alulate the histogram for it. This histogram hols the prevalene of the given hue-saturation omains of the selete region (see Figure 6). (a) Light green (b) Re () Dark blue Figure 6 Histograms mae by the graphial amera toolkit The fifth funtion is to give visual feebak about what objets are reognize an where. For this task there is a eiate winow whih represents the room an whenever an objet is reognize it is rawn on to it. When the amera moves to a new position the previous reognize objets are shae, this way ol reognitions isappear over time. Using this graphial interfae both the amera an the unerlying image proessing faulties an be teste real-time. A printer frienly moifie sreenshot showing the reognize objets mappe to real oorinates is visible in Figure 7, 510

11 Magyar Kutatók 10. Nemzetközi Szimpóziuma 10 th International Symposium of Hungarian Researhers on Computational Intelligene an Informatis this piture illustrates, that the applie amera moel's inverse transformation an aommoate the real sizes an the real shapes of the objets quite well. Figure 7 Image proessing in ation: moifie sreenshot showing reognize objets at the ispae test room in Narvik, Norway Conlusions In this paper we have presente a solution for mobile robot loalization in the Intelligent Spae environment. We showe that it is feasible to ahieve inoor loalization using one pan/tilt surveillane amera an no aitional infrastruture is require. For the objet reognition we use the OpenCV library an the solution is base on olour spae filtering an ontour etetion. The robots o not have to be tagge or arry extra evies. Results an be improve by using multiple ameras with higher resolutions. Conerning future work, we woul like to use a three-amera system, where eah amera overs a smaller area an the image proessing an be one in parallel. Aknowlegement The researh was supporte by ETOCOM projet (TÁMOP /1/KMR ) through the Hungarian National Development Ageny in the framework of Soial Renewal Operative Programme supporte by EU an ofinane by the European Soial Fun an the National Siene Researh Fun (OTKA K62836). Referenes [1] Joo-Ho Lee an Hieki Hashimoto. Intelligent spae onept an ontents. Avane Robotis, 16(3): ,

12 P. Zanaty et al. Image-base Automati Objet Loalisation in ispae Environment [2] Peter Koroni, Hieki Hashimoto, "Intelligent Spae, as an Integrate Intelligent System", Keynote paper of International Conferene on Eletrial Drives an Power Eletronis, Proeeings pp [3] Barry Brumitt, Brian Meyers, John Krumm, Amana Kern, an Steven A. Shafer. Easyliving: Tehnologies for intelligent environments, in Pro. of Seon International Symposium on Hanhel an Ubiquitous Computing, pp , 2000 [4] Larry Ruolph. Projet oxygen: Pervasive, human-entri omputing - an initial experiene. In CAiSE, pages 1 12, 2001 [5] Bra Johanson, Armano Fox, an Terry Winogra. The interative workspaes projet: Experienes with ubiquitous omputing rooms. IEEE Pervasive Computing, 1(2):67 74, 2002 [6] LIREC: LIving with Robots an InterativE Companions, homepage. [Online] [7] Mark Weiser. Some omputer siene issues in ubiquitous omputing. SIGMOBILE Mob. Comput. Commun. Rev., 3(3):12, 1999 [8] Drazen Brsi an Hieki Hashimoto. Moel base robot loalization using onboar an istribute laser range finers, in Pro. of IROS, pp , 2008 [9] Berthol K.P. Horn. Tsai s amera alibration metho revisite. Tehnial report, MIT Artifiial Intelligene Laboratory, 2000 [10] A. D. Millner an M. A. Gooale. The Visual Brain in Ation. Oxfor University Press,