A Fast Multi-camera Tracking System with Heterogeneous Lenses

Transcription

1 213 IEEE/RSJ Internatonal Conference on Intellgent Robots and Systems (IROS) November 3-7, 213. Tokyo, Japan A Fast Mult-camera Trackng System wth Heterogeneous Lenses Xaorong Zhao, Qngy Gu, Tadayosh Aoyama, Takesh Takak, and Idaku Ish Abstract We have developed a fast target trackng system that utlzes four cameras wth lenses of dfferent focal lengths to track an object wthout blurrng mages, even when the object moves n the depth drecton away from the cameras n three-dmensonal (3-D) space. Ths system can mantan a well-focused camera vew by swtchng among the four nput mages, nstead of usng lens motor control. The mult-camera system was mounted on a two-axs mechancal actve vson platform. The actve vson control and camera-vew swtchng are executed by processng color mages from the four camera nputs at 5 fps n real tme on a hgh-speed vson platform. The performance of our system was verfed by ts trackng results for objects movng rapdly n 3-D space. I. INTRODUCTION Recently, real-tme hgh-frame-rate (HFR) vson systems have been developed that can smultaneously process vdeo at hgher frame rates than conventonal vdeo cameras (e.g., NTSC 3 fps) [1] [3]. Such HFR vson systems enable fast target trackng at hundreds of frames per second n real tme for recognton of hgh-speed movng objects. Its effectveness has already been demonstrated n many applcatons such as robot manpulaton [4], vrtual stllness for beatng heart surgery [5], and mcrobe trackng [6]. Several trackng systems wth pattern recognton have been developed by mplementng mage recognton algorthms wth parallel logc [7] [9]. Fast stereo trackng systems usng multple HFR cameras to track an object movng n three-dmensonal (3-D) space have also been reported [1], [11]. Most stereo trackng systems have used lenses wth focal lengths and apertures that are fxed. These systems have a weakness when trackng an object movng n the depth drecton, because the depth of feld (DOF), whch depends on the focal length and aperture of the lens, s also constraned. Autofocus (AF) technology that can automatcally control a motor to poston a lens for well-focused mages s wdely used n consumer dgtal cameras, and many AF algorthms have been proposed [12]. In most AF systems, the focusng speeds are low because the response tmes of the motors that physcally move the varable-focus lenses are long. To shorten the response tme of a varable lens for faster magng, several lqud-flled actve lenses have been developed [13], [14]. However, AF systems that use a sngle camera wth a varable lens have a lmted DOF, because t s dffcult for a sngle varable lens to realze a wde DOF wth hgh mage ntensty. Furthermore, motor mechansms should be ntegrated wth the AF system to control the varable lens, but ther szes and weghts mpose constrants when the AF system s mounted on a movable platform such as a manpulator. In ths study, we develop a mult-camera trackng system that can track a movng object n 3-D space at a hgh frame rate wthout mage blurrng. Our system has four cameras wth lenses of dfferent focal lengths to cover a wde DOF for 3-D trackng. Ths can functon as an AF system wth no lens motor control by selectng well-focused mages from the four cameras. Secton III provdes an outlne of our mult-camera trackng system. Secton III-B descrbes a 3-D trackng algorthm mplemented by our system. Ths algorthm can be executed for mages from four cameras at 5 fps n real tme. Secton IV verfes the realtme 3-D trackng wth several expermental results. II. CONCEPT In our concept of mult-camera trackng, a well-focused camera vew s selected from the vews of multple cameras wth heterogeneous lenses. Ths mnmzes the measurement error caused by lens blur and mage quantzaton when an object s mechancally tracked n a selected camera vew wthout protrudng from the mage sensor area. Fgure 1 llustrates our concept of mult-camera trackng wth heterogeneous lenses when camera DOFs are layered n a mechancal trackng system. The optcal center of the lens of the -th camera ( =1,...,I) les on the plane z =, and the optcal axs s parallel to the z-axs. The (x,y,z) coordnate system s defned as the world coordnate system, and ts z-coordnate represents the depth from the cameras. The lens of the -th camera can be descrbed wth a thn lens model as shown n Fgure 2, where f s the focal length, F s the f-number, a s the dstance from the object to the lens, and b s the dstance from the lens to the mage sensor. A. Measurement error Frst, we consder the mage degradaton when an object s observed wth the -th camera. Dependng on the locaton mul -camera trackng vson Fg. 1. z y x DOF DOF DOF DOF Mult-camera trackng wth heterogeneous lenses /13/$ IEEE 2671

2 object z 1/f =1/a + 1/b mage sensor = 1 = 2 = 1 = 3 = 2 = 3 focal length: f f-number: F lens a Fg. 2. Thn lens model. b Fg = 12 = 23 Relatonshp between measurement error and object locaton. ( ) of the object n the z-drecton, the measurement error by the mage sensor of the -th camera can be descrbed as follows: ε(z)=sgn(z f 2 ( 1 a) F a 1 ) + 2 δ (1) z π where the frst term s the dameter of the crcle of confuson for the -th camera, whch ndcates the lens blur error, and the second term s the quantzaton error for an mage sensor of pxel ptch δ. By multplyng ε(z) wth the rato Z(z)= z b = z ( 1 f 1 a ), (2) the measurement error E(z) of the object can be determned for the -th camera as follows: E(z)= ε(z) Z(z)=sgn(z a) A(z a)+ B, (3) where A and B are constants gven by ( 1 A = f 1 ) f 2 a F a, B = ( 1 f 1 a ) 2 π δ. (4) As expressed n Eq. (3), the measurement error E(z) descrbes a polygonal lne that conssts of two lne segments whose vertex s located at z = a. B. Trackng condton wthout any protruson To track a whole object wth no protruson from the mage, the measurable range n the z-drecton must satsfy the followng condton when an object of wdth L s observed wth the -th camera: 2L z> πn B (= z s ), (5) where N s the number of pxels the mage sensor has n the wdth drecton. C. Selecton of a well-focused camera Under the condton of Eq. (5), the number o (z) of the well-focused camera s determned as follows by searchng for the camera whose measurement error s the smallest: o (z)=argmn E(z), E mn (z)= o(z) E(z), (6) where E mn (z) s the measurement error of the well-focused camera. Fgure 3 shows the relatonshp between the measurement error and thez-coordnate of the object. Here, o (z) and E mn (z) are gven as functons of the z-coordnate. Consderng that the measurement error must be less than a certan value E for accurate observaton, the measurable range n the z-drecton where E mn (z) E holds becomes much wder than the range of z where E(z) E holds for the-th camera. Thus, compared wth sngle-camera trackng, mult-camera trackng wth heterogeneous lenses covers a wder DOF by selectng a well-focused camera dependng on the z-coordnate of a tracked object. D. Advantages and dsadvantages In comparson wth conventonal AF systems that use motor-controlled varable-focus lenses, our concept has two advantages for fast 3-D trackng: (1) t can expand the DOF for 3-D trackng by ncreasng the number of cameras, and (2) t can emulate a fast AF functon wthout any slowresponse motor control. However, our concept also has two dsadvantages for 3-D trackng: (1) a mult-camera system becomes too large and heavy for mountng on a movable platform when the number of cameras becomes too large, and (2) the tracked object s dvergently observed when swtchng between selected camera vews, because the multple cameras are located at dfferent postons owng to ther physcal szes. In ths study, we resolve these dsadvantages by ntroducng a mechancal actve vson platform wth a lghtweght mult-camera head. III. FAST MULTI-CAMERA TRACKING SYSTEM A. System confguraton Based on our concept, we developed a fast mult-camera trackng system. Fgure 4 shows ts confguraton, whch conssts of a two-axs actve vson platform, four camera heads wth lenses of dfferent focal lengths, two FPGA-based mage processng boards (IDP Express board), and a PC. The camera heads and IDP Express boards used n our system were developed for real-tme hgh-speed mage processng [11]. The camera head was compactly desgned for mountng on movable objects. Its dmensons and mass are mm and 145 g, respectvely, when no lens s mounted. The camera head captures 8-bt RGB color mages wth the Bayer flter of ts mage sensor and transfers mages to the IDP Express board at 2 fps. The IDP Express board has two camera nputs and a user-specfc FPGA (Xlnx XC3S5) for hardware mplementaton of hgh-speed mage processng algorthms. The nput 2672

3 four-camera head two-axs actve vson 8-bt RGB mages of pxels IDP Express board 1 5-fps mult-camera trackng for four 512x512 mages Fg. 4. commands torque commands for actve vson System confguraton. hue-based color extracton crcut on a FPGA IDP Express board 2 mages/ processed results personal computer Error [mm] 1E(z) 2E(z) 3E(z) 4E(z) Emn(z) 4 3 = 215 mm = 453 mm = 732 mm z [mm] Fg. 6. Relatonshp between measurement errors and z-coordnate of object for our mult-camera trackng system. Fg. 5. (f=6.mm,f=6) (f=3.5mm,f=6) tlt pan mult-camera head 23mm 23mm two-axs actve vson (f=12.mm,f=6) (f=13.6mm,f=6) A mult-camera head mounted on a two-axs actve vson platform. mages and processng results from two camera heads are transferred va a PCIe 2. bus from the IDP Express board to PC memory at 2 fps. For acceleraton of the multcamera trackng process, we mplemented a hue-based color extracton crcut n the hardware of the user-specfc FPGA on the IDP Express board. Ths crcut extracts specfed hue regons for two color mages and calculates the centrod postons n real tme at 2 fps. In our system, two IDP Express boards were used for the four camera heads, and these two IDP Express boards were mounted on a sngle PC. Through varous API functons assocated wth board control and access to memory-mapped data, arbtrary processes can be programmed as software for realtme executon on the PC. In ths study, we used a PC wth the followng specfcatons: ASUSTeK P5Q-E manboard, Intel Core 2 Quad GHz CPU, 4 GB memory, Wndows 7 (32-bt), and two 16-lane PCIe 2. buses. The four camera heads were mounted on the actve vson platform as a 2 2 mult-camera head. Fgure 5 shows an overvew of the mult-camera head and the actve vson platform. The camera heads are arranged n a 2 2 array wth an nterval of 23 mm, correspondng to the heght and wdth of the camera head. The camera heads were assgned numbers from one to four, proceedng clockwse from the lower left. On the camera heads were mounted Sony NF-mount lenses of dfferent focal lengths: VCL-3S12XM for Camera 1, VCL-6S12XM for Camera 2, and VCL- 3S12XM for Cameras 3 and 4. The dmensons of the multcamera head are mm, and ts mass s 695 g, ncludng the masses of the lenses mounted on the camera heads. For the -th camera ( =1,...,4), the f-number was set as F =6. The focal lengths were set as 1 f =3.5 mm, 2 f =6.mm, and 3 f = 4 f =12.mm. The dstances from the object to the lens were set as 1 a = 165 mm, 2 a = 4 mm, 3 a = 63 mm, and 4 a = 87 mm. For the mult-camera head of our system, Fgure 6 shows the relatonshp between the measurement errors and the z- coordnate of the object. The number o of a well-focused camera s determned as follows: o = 1 (z z 12 ) 2(z 12 <z z 23 ) 3(z 23 <z z 34 ) 4 (z 34 >z), (7) where the z-coordnates at whch to swtch camera vews are z 12 = 215 mm, z 23 = 453 mm, and z 34 = 732 mm. The two-axs actve vson platform s moved by pan and tlt motors, whch are compact hgh-speed RSF-5A-5 motors (Harmonc Drve Systems). The dmensons of the actve vson platform are cm wthout the mult-camera head. The orentaton of the mult-camera head was controlled usng the vsual nformaton from the camera heads n order to track a movng object n 3-D space. B. The mplemented algorthm For fast trackng of a colored object movng n 3-D space, our system mplements a mult-camera trackng algorthm, whch has the sx steps descrbed below. Fgure 7 shows the flowchart of the algorthm mplemented. (1) Hue-based color regon extracton To extract a specfed color regon from an mage as an object to be tracked, each color nput mage from the -th camera ( =1,,4) s assgned thresholds n the HSV color space [15] as follows: { ( ) 1 B(X,Y)= H H <θ H, S>θ S, V>θ V. (8) (otherwse) 2673

4 hardware mplementa on halted next frame Fg x512 5 fps for 4 camera heads (1) hue-based color regon extrac on (2) mage centrod calcula on mage centrods for selected two cameras (3) border lne nspec on (4) stereo-based 3-D calcula on 3-D pos on (5) well-focused camera selec on mage centrod of well-focused camera (6) ac ve vson control bnary mages mage centrods 3-D pos on camera number 512x512 mage pan/ lt angles Flowchart of mult-camera trackng algorthm. where H(X,Y), S(X,Y), and V(X,Y) are the hue, saturaton, and value components of the nput mage of the -th camera; H, θ H, θ S,andθ V are parameters to specfy a vvd color that corresponds to the object to be tracked. The observatonal ranges of hue, saturaton, and value are set to H<36, S 1, and V 255, respectvely. The HSV color mages are converted nto 8-bt mages after RGB converson of the nput mages wth a Bayer flter. (2) Image centrod calculaton For the -th camera, the zeroth-order and frst-order moment features of B(X,Y) are computed as follows: M = X,Y M X = X,Y B(X,Y), (9) X B(X,Y), M Y = X,Y Y B(X,Y). (1) The poston of a tracked object n the nput mage of the -th camera s calculated as the followng mage centrod: c =( c X, M X c Y )=( ) M Y, M. (11) M (3) Borderlne nspecton On the borderlnes of a mage, the number of pxels where B(X,Y)=1s counted for the -th camera as follows: n b = B(X,Y), (12) (X,Y) X b where (X,Y) X b ndcates that (X,Y) s located on X =, X = 511, Y =,ory = 511. IfN b s defned as the set of the numbers of the camera heads for whch no pxel of the specfed color les on the borderlnes of the mages, then Camera s and Camera s 1 are selected as a par of camera heads for a stereo-based 3-D calculaton, as follows: s =max N b, N b = { : n b =}. (13) The trackng process s halted when N b = or s =1but s restarted when the object to be tracked s observed wth two or more camera heads. (4) Stereo-based 3-D calculaton The 3-D poston of the tracked object s calculated by trangulaton as follows: (x,y,z)=stereo s s 1( s c, s 1 c). (14) where Stereo j ( ) s a trangulaton functon to calculate the 3-D poston of the object from correspondng ponts n astereoparofcamera and Camera j. All the camera parameters are ntally obtaned from a pror calbraton. (5) Well-focused camera selecton Dependng on the z-coordnate, the number o of a wellfocused camera s selected n accordance wth Eq. (7). (6) Actve vson control The pan and tlt motors of the two-axs actve vson platform are controlled to mnmze the followng evaluaton functon: 4 F = w c x 2, (15) =1 where F s the weghted sum of the squared errors between the mage centrods and the center pxels (256,256) of the mages from the four camera heads. The weght w ( =1,,4) s determned usng the z-coordnate of the tracked object as follows: ( (1,,,) (z 1 D) 2 ) D z 2 D 1 D, z 1 D 2 D 1 D,, ( 1 D<z 2 D) ( ) (w 1,w 2,w 3,w 4 )=, 3 D z 3 D 2 D, z 2 D 3 D 2 D, ( 2 D<z 3 D), (16) (,, 4 D z 4 D 3 D, z 3 D 4 D 3 D ) ( 3 D<z 4 D) (,,,1) ( 4 D<z) where the D ( =1,,4) are determned by usng z 12, z 23,andz 34 n the formulas 1 D =(3z 12 z 23 )/2=96mm, (17) 2 D =(z 12 +z 23 )/2 = 334 mm, (18) 3 D =(z 23 +z 34 )/2 = mm, (19) 4 D =( z 23 +3z 34 )/2 = mm. (2) In ths study, Steps 1 and 2 were mplemented by parallel hardware logc for 8-bt color mages on the user-specfc FPGA of the IDP Express, and Steps 3 6 were mplemented by software on the PC. We have confrmed that all sx steps can be executed for mages n real tme at 5 fps, and so the motors of the actve vson platform were controlled usng vsual feedback at 5 Hz. IV. EXPERIMENTS We present trackng results for an object moved by a human hand. Fgure 8 shows the expermental setup. In the experments, an artfcal red rose flower of 6 6 7mm sze was moved wth respect to a color-patterned background. To extract the red regon, the parameters for bnarzaton were set to θ S =.3, θ V = 76, θ H = 16.9, and H =

5 6 x y z ar fcal flower y z x 4 camera heads x,y,z [mm] (a) 3-D poston 2-axs ac ve vson Fg. 8. Expermental setup. t =.46 s t =.49 s t =.52 s t =.55 s t =.58 s t =.61 s (b) mages from Camera 2 A. Rotaton at fxed depth To demonstrate the trackng performance of our multcamera system wth the four camera heads mounted, we conducted a trackng experment n whch an artfcal flower was manually rotated 35 cm n front of the mult-camera head at fve rotatons per second. Fgure 9(a) shows the observed (x,y,z) coordnates of the tracked object n the camera coordnate system over an nterval of 2 s startng at t =.Thez-coordnate vared slghtly around 35 mm wthn a range of z 12 = 215 mm and z 23 = 453 mm, so Camera 2 was always selected as the wellfocused camera among the four camera heads. Fgure 9(b) shows the nput sequence of sx mages captured by Camera 2 at ntervals of.3 s for t = s. Fgure 9(c) shows the (X,Y) coordnates of the mage centrod of Camera 2 over the nterval of 2 s startng at t =. It can be observed n Fgures 9(b) and 9(c) that the mage centrod of Camera 2 was always n the camera vew to be controlled near the center pxel (256,256), but there were devatons caused by the lmtatons n the abltes of the motors of the actve vson platform. Owng on these devatons, the (x,y) coordnates of the tracked object n the camera coordnate system were also devated n Fgure 9 (a). Fgure 9(d) shows the pan and tlt angles of the actve vson platform over the nterval of 2 s startng at t =. It can be observed that the pan and tlt angles vared perodcally at fve cycles per second, correspondng to the rotaton of the artfcal flower. B. Back-and-forth moton n the depth drecton To demonstrate the camera selecton performance of our mult-camera system, we conducted a trackng experment n whch an artfcal flower was manually moved back and forth between 2 cm and 12 cm n front of the mult-camera head at two cycles per second. Fgure 1(a) shows the (x,y,z) coordnates of the tracked object and the number of the well-focused camera over an nterval of 2 s startng at t =. It can be observed that the z-coordnate vared perodcally at 2 cycles per second, whereas the (x,y) coordnates were relatvely constant. The number of well-focused camera was selected accordng to the z-coordnate and became smaller when the tracked object became nearer to the camera head. Fgure 1(b) shows the X,Y [pxel] pan, lt [deg] X () Y () (c) mage centrod of Camera Fg. 9. (d) pan and tlt angles of actve vson platform Results obtaned when a flower was rotated at fxed depth. nput sequences of sx mages captured by the four camera heads at ntervals of.3 s for t = s. The mages wth red borders were selected as the well-focused camera vews wthout any protruson of the artfcal flower from the mage. Fgure 1(b) also shows the sze-normalzed mages that were generated by reszng and croppng the selected well-focused mages so that the artfcal flower was located at the center of each mage and was adjusted n sze to 35% of the pxels. Thus, the tracked object was always located at the center of the sze-normalzed mage. Fgure 1(c) shows the (X,Y) coordnates of the mage centrods of the four camera heads over the nterval of 2 s startng at t =. Here the thck lne ndcates the mage centrod of the well-focused camera. It can be observed n Fgures 1(b) and 1(c) that the mage centrod of the wellfocused camera was near the center pxel (256,256), but there were devatons when the well-focused camera was swtched accordng to the z-coordnate of the tracked object. Fgure 1(d) shows the pan and tlt angles of the actve vson platform over the nterval of 2 s startng at t =. The pan lt 2675

6 pan and tlt angles changed at the moment that the wellfocused camera was swtched to reflect the moton of the artfcal flower n the depth drecton, but otherwse these angles vared lttle. Ths s because the pan and tlt angles were controlled to reduce the devaton of the mage centrod when the well-focused camera was swtched. The expermental results show that our system can correctly track an object n a well-focused mage, even when the object moves rapdly n 3-D space. Our system acheves ths by processng the mages from the four camera heads at 5 fps for actve vson control and selecton of the well-focused camera. V. CONCLUSION We have developed a fast mult-camera trackng system usng four cameras wth heterogeneous lenses for mechancally trackng an object movng n 3-D space. By swtchng among the four cameras to mantan a well-focused camera vew, ths system can track an object n mages at 5 fps wthout blurrng. Its effectveness was verfed by ts trackng results for objects movng rapdly n 3-D space. On the bass of these results, we wll mprove our system for hgh-speed 3-D robot control n the real world by expandng the mage feature extracton for heterogeneous stereo cameras. REFERENCES [1] Y. Watanabe, T. Komura, and M. Ishkawa, 955-fps real-tme shape measurement of a movng/deformng object usng hgh-speed vson for numerous-pont analyss, Proc. IEEE Int. Conf. Robot. Automat., , 27. [2] S. Hra, M. Zakoj, A. Masubuch, and T. Tsubo, Realtme FPGAbased vson system, J. Robot. Mechat., 17, 41 49, 25. [3] I. Ish, T. Tanguch, R. Sukenobe, and K. Yamamoto, Development of hgh-speed and real-tme vson platform, H 3 vson, Proc. IEEE/RSJ Int. Conf. Intell. Rob. Sys., , 29. [4] A. Namk, Y. Ima, and M. Ishkawa, Development of a hgh-speed multfngered hand system and ts applcaton to catchng, Proc. IEEE/RSJ Int. Conf. Intell. Rob. Sys., , 23. [5] Y. Nakamura, K. Ksh, and H. Kawakam, Heartbeat synchronzaton for robotc cardac surgery, Proc. IEEE Int. Conf. Robot. Automat., , 21. [6] H. Oku, I. Ish, and M. Ishkawa, Trackng a protozoon usng hghspeed vsual feedback, Proc. Annu. Int. IEEE-EMBS Specal Topc Conf. Mcrotechnologes n Medcne and Bology, , 21. [7] I. Ish, T. Ichda, Q. Gu, and T. Takak, 5-fps face trackng system, J. Real-Tme Image Proc., do: 1.17/s , 212. [8] I. Ish, T. Tatebe, Q. Gu, and T. Takak, Color-hstogram-based trackng at 2 fps, J. Electron. Imagng, 21, 131, 212. [9] Q. Gu, T. Takak, and I. Ish, Fast FPGA-based mult-object feature extracton, IEEE Trans. Crcuts and Syst. Vdeo Technol., 23, 3 45, 213. [1] Y. Nakabo, I. Ish, and M. Ishkawa, 3D trackng usng two hghspeed vson systems, Proc. IEEE/RSJ Int. Conf. Intell. Rob. Sys., , 22. [11] I. Ish, T. Tatebe, Q. Gu, Y. Morue, T. Takak, and K. Tajma, 2 fps real-tme vson system wth hgh-frame-rate vdeo recordng, Proc. IEEE Int. Conf. Robot. Automat., , 21. [12] L. Shh, Autofocus survey: a comparson of algorthms, Proc. SPIE Electron. Imagng, 6528, 27. [13] V. Laude and C. Drson, Lqud-crystal actve lens: applcaton to mage resoluton enhancement, Opt. Comm., 163, 72 78, [14] H. Oku and M. Ishkawa, Hgh-speed lqud lens wth 2 ms response and 8.3 nm root-mean-square wavefront error, Appl. Phys. Lett., 94, 22118, 29. [15] A. R. Smth, Color gamut transform pars, Proc. SIGGRAPH Comput. Graph., 3, 12 19, x,y,z [mm] X [pxel] Y [pxel] pan, lt [deg] x y z number sze-normalzed mage me[s] (a) 3-D poston and number of a well-focused camera t = 1.59 s t = 1.62 s t = 1.65 s t = 1.68 s t = 1.71 s t = 1.74 s (b) nput mages and sze-normalzed mages well-focused camera me[s] me[s] (c) mage centrods (d) pan and tlt angles of the actve vson platform Fg. 1. Results obtaned when a flower was moved back-and-forth n the depth drecton. pan lt 2676