An Active Stereo Vision System Based on Neural Pathways of Human Binocular Motor System

Transcription

1 Journal of Bionic Engineering 4 (2007) An Active Stereo Viion Sytem Baed on Neural Pathway of Human Binocular Motor Sytem Yu-zhang Gu 1, Makoto Sato 2, Xiao-lin Zhang 2 1. Department of Advanced Applied Electronic, Tokyo Intitute of Technology, 4259 Nagatuda, Midori-ku, Yokohama, Japan 2. Preciion and Intelligence Laboratory, Tokyo Intitute of Technology, 4259 Nagatuda, Midori-ku, Yokohama, Japan Abtract An active tereo viion ytem baed on a model of neural pathway of human binocular motor ytem i propoed. With thi model, it i guaranteed that the two camera of the active tereo viion ytem can keep their line of ight fixed on the ame target object during mooth puruit. Thi feature i very important for active tereo viion ytem, ince not only 3D recontruction need the two camera have an overlapping field of viion, but alo it can facilitate the 3D recontruction algorithm. To evaluate the effectivene of the propoed method, ome oftware imulation are done to demontrate the ame target tracking characteritic in a virtual environment apt to mitracking eaily. Here, mitracking mean two eye track two different object eparately. Then the propoed method i implemented in our active tereo viion ytem to perform real tracking tak in a laboratory cene where everal peron walk elf-determining. Before the propoed model i implemented in the ytem, mitracking occurred frequently. After it i enabled, mitracking never occurred. The reult how that the viion ytem baed on neural pathway of human binocular motor ytem can reliably avoid mitracking. Keyword: binocular motor ytem, neural pathway, gaze, mitracking Copyright 2007, Jilin Univerity. Publihed by Elevier Limited and Science Pre. All right reerved. 1 Introduction For an active tereo viion ytem, the cooperative camera movement are very important. Generally, two camera gazing at the ame patial point can provide an overlapping field of viion and mall tereo diparitie around the gaze point, which make tereo fuion and 3D recontruction eaier. Conventionally, image proceing approache are applied to make ure that the two camera are tracking the ame target object. There are many method propoed to olve thi control problem. Coomb propoed a method called Zero Diparity Filter (ZDF) which can iolate the target object from a ditracting background [1]. It i an effective method to obtain the target poition in a imple cene, but it tend to output a lot of fale zero diparitie in a complicated cene, ince ZDF i jut the logic AND of tereo vertical edge image. Rougeaux propoed a method which can obtain the location of the target moving acro the horopter by uing ZDF and virtual horopter [2]. Becaue it relie on ZDF, it i apt to fail in complicated cene alo. Tanaka propoed a diparity baed egmentation method to reduce the fale output of ZDF [3], but it cannot reolve the problem that there are many object on the horopter. On the whole, ome of them can make good performance only when the cene i imple, other are deigned to adapt certain pecial cene. Evidently, it i inufficient to olve the problem only with image proceing approache. It i well known that the ocular motor control ytem i an effective device for capturing an object in the central pit of the retina. Human eye alway orientate Correponding author: Yu-zhang Gu gu.y.ab@m.titech.ac.jp

2 186 Journal of Bionic Engineering (2007) Vol.4 No.4 their line of ight to keep the image of the object from which are decribed in our previou work in detail [5]. leaving their central pit, i.e. eyeball never let an object image moving freely on their retina. Thi i the motivation that we develop an active tereo viion ytem baed on human binocular motor ytem model. In thi paper, firt we briefly introduce our mathematical model derived from the neural pathway of human binocular motor ytem propoed in our previou work. Second, we do ome oftware imulation to optimize the parameter of our control model, a well a exhibit the unique feature. Then, we introduce our active tereo viion ytem which i made of two PTZ camera, Fig. 1 Horizontal binocular motor ytem model. one omnidirectional camera, and one peronal computer. The model propoed in thi paper i a implified Finally, we launch it in our laboratory where everal verion of the previou one. The block of vetibular peron walk elf-determining to tet the robutne of nucleu i eraed. It mean that the model i appropriate the ytem in a relatively complicated environment. for head-fixed binocular motor ytem, rather than that 2 Mathematical model mounted on a vehicle or ego-motion robot. The implification facilitate u to contruct a real ytem, and it doe not affect the eential binocular cooperative movement character of the original model. According to Fig. 1, Eq. (1) and Eq. (2) can be Eye movement include horizontal rotation and vertical rotation. Since two eye are arranged horizontally, to keep gazing on a moving object the two eye have to do horizontal conjugate movement or vergence movement from time to time. Therefore the horizontal binocular eye movement control i more complex than that of vertical binocular eye movement. Here we propoe the horizontal mathematical model derived from our previou work [4,5]. For vertical binocular eye movement control, jut a few modification of the horizontal model are neceary. Becaue of the complexity of human being ocular motor ytem, ome aumption are introduced to implify the model. Firt, all the tranfer function in the ocular motor ytem model are conidered linear [6 8]. Second, delay caued by image proceing in the viual cortex and ignal tranmiion in the nerve are diregarded. Third, cognition in the cerebrum, including ynthetic inference, prediction, and the influence of attention [9] i neglected. The propoed mathematical model i hown in Fig. 1. φ l and φ r are repectively deired value to fix the line of ight of left and right eye onto the viual target. E l and E r are repectively the controlled rotation angle of left and right eye. σ, v, κ, η, ρ 1 and ρ 2 are gain modeled from neural pathway of human binocular motor ytem, obtained, ρ1 El() = ( σ + ν )( ϕ l() El()) + ( κ + η )( ϕ l( ) El( )) + ( σ + ν )( ϕ r( ) Er( )) ρ1 + ( κ + η )( ϕ r( ) Er( )), (1) ρ1 Er() = ( κ + η )( ϕ l() El()) + ( σ + ν )( ϕ l( ) El( )) + ( κ + η )( ϕ r( ) Er( )) ρ1 + ( σ + ν )( ϕ r( ) Er( )). (2) By umming up Eq. (1) and Eq. (2), Eq. (3) can be obtained, ( σ + ν + κ + η)( ϕl( ) + ϕr( )) El() + Er() =. (3) 1 σ + κ + ( + ν + η) ρ + ρ

3 Yu-zhang Gu et al.: An Active Stereo Viion Sytem Baed on Neural Pathway of Human Binocular Motor Sytem 187 By ubtracting Eq. (2) from Eq. (1), Eq. (4) can be The different time contant between conjugate and obtained, vergence eye movement can efficiently prevent binocular motor ytem from mitracking. In thi paper, ( σ + ν κ η)( ϕl( ) ϕr( )) El() Er() =. (4) 1 σ κ + ( + ν η) mitracking mean two eye track two different object ρ1 eparately. From Eq. (3) and Eq. (4), Eq. (5) can be obtained, El () 1 σ + κ + ( ν + η) ( ϕl( ) + ϕr( ) E () 2 1 r = ( ϕ ( ) ( ) ( ) l ϕr + σ + κ + + ν + η ρ1+ 1 σ κ + ( ν η) ( ϕl( ) ϕr( ) ( ϕ ( ) ( ) ( ) r ϕl σ κ + + ν η ρ ρ (5) Obviouly, the firt part of the right ide of Eq. (5) i the tranfer function of conjugate eye movement, and the econd part i the tranfer function of vergence eye movement. The time contant of the conjugate tranfer function T c and that of the vergence tranfer function T v can be obtained, a hown in Eq. (6) and Eq. (7). 1 + ( ρ1+ )( ν + η) T c =, (6) ( ρ + ρ )( σ + κ) T v 1 + ( ρ1 )( ν η) =. (7) ( ρ ρ )( σ κ) To make ure that the ytem i table, condition Eq. (8) or Eq. (9) hould be met [5] σ > κ, ν > η, ρ > ρ, (8) σ < κ, ν < η, ρ < ρ. (9) Either Eq. (8) or Eq. (9) make T c > T v. In other word, in a table ytem the time contant of vergence eye movement i longer than that of conjugate eye movement. Thi concluion accord with normal phyical phenomenon. When the viual target move in a plane perpendicular to the line of ight approximately, it image in the retina move quickly. Short conjugate eye movement time contant enable eye rotation to repond quickly to keep mooth puruit. In cae of the viual target move along the line of ight, it image in the retina move lowly. Even though the time contant of vergence eye movement i longer, the two eye till can follow the viual target. 3 Mitracking problem Uually mitracking occur when an object imilar to the viual target object being tracked croe the target object. The reaon of mitracking i that the ytem cannot ditinguih two imilar object correctly. Of coure, killful image proceing algorithm can improve the ability of ditinguih imilar object to a certain extent. For example, we can calculate the velocity vector of target object. But in a real complicated cene, thi kind of proceing may become very unreliable. It i not a practicable approach to olve the mitracking problem through complicated image proceing algorithm only. The character of the propoed binocular motor ytem model can be ued to olve the mitracking problem. A decribed in ection 2, the time contant of vergence eye movement i longer than that of conjugate eye movement. Thi character provide u a way to prevent the ytem from mitracking. In Fig. 2, there i an object B imilar to the tracked target object A. At firt the two camera track object A. When object A and object B cro each other, the image of them are too imilar to be ditinguihed. Suppoe that the left camera keep tracking object A. But the right camera i confued by imilar object A and B, and it trie to track object B. At thi time mitracking occur, and the ytem witche from the original conjugate eye movement to vergence eye movement. Becaue of the longer time contant of vergence eye movement, neither Fig. 2 Mitracking problem.

4 188 Journal of Bionic Engineering (2007) Vol.4 No.4 of the eye can follow it own viual target object. Accordingly, the image of the viual target object will depart from the center of camera enor quickly. In other word, the abrupt departure indicate that mitracking may occur. At thi time both object A and B are till near the gazing point. Now if the ytem earche all poible viual target object nearby the center of camera enor immediately, conidering that the binocular diparity of object near the gazing point i mall enough, Fig. 4 Vergence tep repone. the ytem can review whether the two camera are attempting to track different object, and return to normal tracking condition. In next ection, ome oftware imulation experiment are done to verify the mitracking prevention effect of the propoed binocular motor ytem model. 4 Software imulation Theoretically, to track a target a fat a poible, both T c and T v hould be kept a horter a poible. But in nature, a decribed in ection 2, fat vergence eye movement i not neceary. According to the table condition requirement decribed in Eq. (8), we propoe a et of effective parameter a σ = 3, v = 0.5, κ = 2.5, η = 0.2, ρ 1 = 1.5 and ρ 2 = 0.5. According to Eq. (6) and Eq. (7), the conjugate time contant i T c = 0.191, and the vergence time contant i T v = 2.6. Fig. 3 and Fig. 4 how the tep repone of conjugate eye movement and vergence eye movement of thi model. Suppoe there are two object A and B move horizontally in front of the binocular motor ytem. The ditance from the object to the bae point of the binocular motor ytem i far longer than that between the two camera. At firt the two camera of the ytem track object A. When object A and B cro each other, the left camera keep track object A, but the right camera begin to track the other object B (a hown in Fig. 2). Fig. 5 how the imulation reult, in which both object A and object B move toward each other at peed of 5 deg 1 (relative peed i 10 deg 1 ). At time t = 3, A and B cro each other, and mitracking occur. For the reaon explained in ection 3, neither of the camera can follow it own viual target object. Fig. 5 Mitracking imulation reult. Fig. 3 Conjugate tep repone. Let D t denote the binocular diparity of target, D a denote binocular diparity of object A, and D b denote binocular diparity of object B. Eq. (10), Eq. (11) and Eq. (12) can be given, D = ( θ θ ) ( θ θ ), (10) t left_target left_eye right_target right_eye

5 Yu-zhang Gu et al.: An Active Stereo Viion Sytem Baed on Neural Pathway of Human Binocular Motor Sytem 189 Da = ( θobject_a θleft_eye ) ( θobject_a θright_eye ) (11) = θright_eye θleft_eye, Db = ( θobject_b θleft_eye ) ( θobject_b θright_eye ) (12) = θright_eye θleft_eye. Fig. 6 how the change of D t and D a (D b ) during the tracking proce of Fig 5. When mitracking occur at time t = 3, the binocular diparity of the viual target increae dramatically. The increae in D t can be treated a a ignal which indicate that mitracking ha occurred. although it may not be the correct one. Thi mean that, becaue D a equal D b theoretically, the model cannot enure that the ytem till track object A, even though it can prevent ytem from mitracking. To illutrate the unique effect of our model, the imulation reult of a binocular ytem with two independent control block for it two camera i given in Fig. 7. Suppoe the independent control block i a firt order lag tranfer function with gain = 1 and time contant = Fig. 6 Binocular diparity. Table 1 how the relation between D t and D a (D b ) when mitracking occur. Table 1 D t and D a in Fig. 6 t () D t ( ) D a ( ) Fig. 7 Mitracking imulation reult (without propoed model). Obviouly, the two camera can keep tracking their own viual target even if the right camera confued object B with object A after time t = 3. Fig. 8 how the change of D t and D a (D b ) during tracking proce of Fig. 7. Obviouly, the abolute value of D t i greater than that of D a (D b ) after mitracking. That mean if the ytem compare current binocular diparity of the target with all poible binocular diparitie of other object near the gazing point when mitracking occur, it can ditinguih different object and track only one of them Fig. 8 Mitracking imulation reult (without propoed model).

6 190 Journal of Bionic Engineering (2007) Vol.4 No.4 Similarly, Table 2 how the relation between D t and D a (D b ) when mitracking occur in Fig. 8. Table 2 D t and D a in Fig. 8 t () D t ( ) D a ( ) Fig. 10 Binocular diparity (v = 3 deg 1 ) Unlike that hown in Fig. 6, in D t Fig. 7 only increae a little oon after mitracking. Then D t keep at a low value in the ret of the tracking proce. Theoretically, a little change of D t at time t = 3 can be treated a a ignal indicating mitracking. But in practice, uch a low level ignal i too weak to be picked up reliably from noie. Mitracking i unavoidable once it tarted in uch kind of ytem. The imulation mentioned previouly i an example to explain the mechanim of how the propoed model can prevent mitracking from occurring. In practice, object A and B can cro with each other at different relative peed. Fig. 9, Fig. 10 and Fig. 11 how the relation between D t and D a (D b ) when A and B cro each other at different relative peed v. Fig. 9 Binocular diparity (v = 1 deg 1 ). Fig. 11 Binocular diparity (v = 5 deg 1 ). Obviouly, at different relative peed v, the increae tyle of D t and D a i the ame, and the increae peed of D t and D a i in direct proportion to the relative peed of object A and B. There are two thing that are independent to the relative peed v. Firt, D t will keep increae after mitracking. Second, the abolute value of D t i greater than that of D a all the time after mitracking. Therefore, in real ytem a relatively low threhold, which i lightly greater than it noie level, can be et to detect mitracking. When D t get greater than thi threhold, mitracking i detected and the ytem will get a chance to review it viual target. Becaue D a < D t, at leat one of object A and B i cloe to current gazing point, o the ytem i able to find out the correct viual target eaily and return to track it immediately. 5 Experiment ytem An active tereo viion ytem i developed to implement the propoed binocular motor ytem model. Fig. 12 how the active tereo viion ytem developed. The main part of the ytem are two Canon VC-C50i PTZ camera. VC-C50i output NTSC model video

7 Yu-zhang Gu et al.: An Active Stereo Viion Sytem Baed on Neural Pathway of Human Binocular Motor Sytem 191 ignal, and it 1/4 inch CCD enor ha 340,000 effective pixel. The focal length i from 3.5 to 91.0 mm, providing 26 power zoom. The pan rotation angle range i left 100 to right 100, and rotation peed i 1 to 90 per econd. The tilt rotation angle range i up 90 to down 30, and rotation peed i 1 to 70 per econd. There i an omnidirectional camera between the two PTZ camera. Thi camera i utilized to monitor a wider cene, ince the VC-C50i can only provide maximal 40 angle of view. To real-time proce image from three camera, we ue a peronal computer with a 3.2 GHz Pentium D CPU. Fig. 12 Active tereo viion ytem. Image from the omnidirectional camera are ued to detect moving object in the monitored cene. A imple background ubtraction algorithm i implemented to do moving object detection. When a moving object i detected, the two PTZ camera are rotated to make the image of the moving object hift to the enor center. Then the ytem detect the face from the image of the two PTZ camera. We implement booted cacade of imple feature object detection algorithm to detect face [10,11]. After the face wa detected, the two PTZ camera are rotated to make the image of the detected face hift to the enor center. Then the ytem begin to moothly puruit the moving face. At firt we tried to ue dynamic updating template matching method to track the viual target, but we found it i not reliable enough. Finally we adopted Continuouly Adaptive Mean Shift (CAMSHIFT) method [12], which i more effective and robut to noie. We do an experiment to track peron walking in our laboratory. The background i relatively complicated and there are everal peron to confue the ytem. Becaue kin color i the only feature ued for object tracking, the ytem i eaily confued when two face cro each other in front of it. Before the propoed model i implemented in the ytem, mitracking occurred frequently. After we enabled the propoed model, mitracking never occurred. The reult how that the viion ytem baed on neural pathway of human binocular motor ytem can reliably avoid mitracking problem. The video clip can be found in our web ite Concluion In thi paper, an active tereo viion ytem baed on a model of neural pathway of human binocular motor ytem i propoed. The model include conjugate eye movement and vergence eye movement. The vergence time contant i far greater than the conjugate time contant. By imulation, we found thi feature can guarantee that the two camera of the active tereo viion ytem can keep their line of ight fixed on the ame target object during tracking. Thi feature i very important for active tereo viion ytem, ince not only 3D recontruction need the two camera have an overlapping field of viion, but alo it can facilitate the 3D recontruction algorithm. Reference [1] Coomb D J, Brown C M. Cooperative gaze holding in binocular viion. IEEE Control Sytem Magazine, 1991, 11, [2] Rougeaux S, Kita N, Kuniyohi Y, Sakane S. Tracking a moving object with a tereo camera head. Proceeding of the 11th Annual Conference of Robotic Society of Japan, Japan, 1993, [3] Tanaka M, Maru N, Miyazaki F. Binocular gaze holding of a moving object with the active tereo viion ytem. Proceeding of the Second IEEE Workhop on Application of Computer Viion, Saraota, FL, USA, 1994, [4] Zhang X, Wakamatu H. A unified adaptive oculomotor control model. International Journal of Adaptive Control and Signal Proceing, 2001, 15, [5] Zhang X, Wakamatu H. Mathematical model for binocular movement mechanim and contruction of eye axe control ytem. Journal of the Robotic Society of Japan, 2002, 20,

8 192 Journal of Bionic Engineering (2007) Vol.4 No ology, 1996, 76, [6] Robinon D A. The ue of control ytem analyi in the [10] Viola P, Jone M. Rapid object detection uing a booted neurophyiology of eye movement. Annual Review of cacade of imple feature. IEEE Computer Society Conference on Computer Viion and Pattern Recognition, Ha- Neurocience, 1981, 4, [7] Galiana H L, Outerbridge J S. A bilateral model for central waii, USA, 2001, neural pathway in vetibuloocular reflex. Journal of Neurophyiology, 1984, 51, for rapid object detection. IEEE International Conference on [11] Lienhart R, Maydt J. An extended et of haar-like feature [8] Raphan T, Wearne S, Cohen B. Modeling the organization of Image Proceing, Rocheter, New York, 2002, the linear and angular vetibulo-ocular reflexe. Annal of [12] Bradki G R. Computer viion face tracking for ue in a the New York Academy of Science, 1996, 781, perceptual uer interface. Intelligence Technology Journal, [9] Uchino Y, Saaki H, Sato H, Imagawa M, Suwa H, Iu N. 1998, 2, Utriculoocular reflex arc of the cat. Journal of Neurophyi-