Application of Visual Tracking for Robot-Assisted Laparoscopic Surgery

Transcription

1 Applcaton of Vsual Trackng for Robot-Asssted Laparoscopc Surger Xaol Zhang and Shahram Paandeh Expermental Robotcs Laborator School of Engneerng Scence Smon Fraser Unverst Burnab, BC, V5A 1S6, Canada Abstract: Wth the ncreasng populart of laparoscopc surger, the demand for better modes of laparoscopc surger also ncreases. The current laparoscopc surger mode requres an assstant to hold and manpulate the endoscope through commands from the surgeon. However, durng length surger procedures, accurate and on tme adustment of the camera cannot be guaranteed due to the fatgue and hand tremblng of the camera assstant. Ths paper proposes a practcal vsual trackng method to acheve automated nstrument localzaton and endoscope maneuverng n robot-asssted laparoscopc surger. Solutons concernng ths approach, such as, endoscope calbraton, marker desgn, dstorton correcton and endoscope manpulator desgn are descrbed n detal. Expermental results are presented to show the feasblt of the proposed method. Kewords: Laparoscopc Surger, vsual trackng, calbraton, barrel dstorton 1. Introducton: Recentl, laparoscopc surger ganed ncreasng populart because of ts mnmall nvasve nature. In laparoscopc surger, a surgcal operaton can be performed wth an endoscope and several long, thn, rgd nstruments (see Fgure 1) through several small Camera Endoscope Instruments Fgure 1. Tools needed for laparoscopc surger. ncsons. Compared wth tradtonal surger, laparoscopc surger provdes less pan, faster recover and small nures. However, to control the surgeon s feld of vew, an assstant surgeon s needed to hold and manpulate the camera. Durng length

2 procedures, accurate and on tme adustment of the camera cannot be guaranteed due to the fatgue of the camera assstant. Instead, usng a robot to control the camera wll result n less erroneous camera moton and can off-load routne tasks. A tpcal laparoscopc surger usng robot camera assstant s shown n fgure 2. The endoscope relas mages of nternal organs to a camera, and the mages are dsplaed on a vdeo screen placed n front of the surgeon. However, the lack of ntellgence n the Montor for vewng the operaton ste Robot Assstant Fgure 2. Laparoscopc surger usng a robot camera assstant. desgn of current robot camera controllers presents a challengng problem. For example, the surgeon n general s often dstracted b ssung postonng command to the robot manpulator va hand-held controllers, a foot pedal or other nterface such as voce control, e.g., voce controlled robot AESOP[1]. To mprove the current robot-asssted laparoscopc surger sstem, we propose to use vsual trackng technques to enable automatc robotcall control of the camera. In order to track the nstrument, we need to know the three-dmensonal (3D) postonal nformaton. In general, such 3D nformaton can be obtaned from stereo-endoscop (the use of two separate optcal channels, each wth ts own lens optcs). However, ths approach doubles the hardware, thus t s not wdel accepted b clncans. We propose a method that uses a sngle endoscope to obtan the 3D postonal nformaton of the nstrument. Vsual trackng s an actve research area addressed b man researchers. Prewer and Ktchen [2] used edge correspondence to track the mage features. Azarbaehan and Pentland [3] obtaned the 3D locatons of the face and hands b trackng skn blobs. Redd and Lozou [4] used optcal flow feld to control the camera s moton to keep a target at the center of the camera s feld of vew. B usng pror knowledge about how humans move, Leventon [5] bult a Gaussan probablt model to reconstruct the 3D moton of a human fgure from a monocular mage sequence. However, onl a few researchers addressed vsual trackng n laparoscopc surger: Wang et al. [6] used color statstcs to classf, group and label nstrument drectl, We et al. [7] desgned a marker and usng color selecton and color segmentaton to detect and locate the marker. Due to the dverst of the nstruments, t s hard to track the tp of nstruments drectl as descrbed n [6]. In ths paper, we propose to desgn a marker and attach the marker to the tp of the nstrument. When needed, vsual trackng technques can be

3 appled to track the marker as the nstrument moves. Instead of usng color nformaton for vsual trackng as proposed n [6] and [7], we use monochrome data for processng. Generall, monochrome mage processng s qucker, smpler and robust. The 3D postonal nformaton of the marker then wll be used to realze automatc trackng of the nstruments. There are man dffcultes assocated wth usng markers for nstrument localzaton. Frst, mages taken from the endoscope are dstorted so t s dffcult to obtan the correct measurement of the mages. Second, tools ma enter the endoscope feld of vew from dfferent drectons and at varous angles. Fnall, usng mages from mono-endoscope to reconstruct 3D spatal nformaton s a challengng task. These problems wll be addressed and solved n later parts of ths paper. straght vewng end 0 o lght cable connecton vew angle (a) vew angle rod lens (b) oblque vewng end (c) Fgure 3. Rod Lens endoscope (a) An endoscope featurng a conventonal lens sstem. Refracton occurs as lght passes through small lenses seated at dstant ntervals. (b) A HOPKINS rod lens endoscope. The rod lenses are seated at close ntervals, resultng n ο better lght transmsson and reduced dameter. (c) Forward-oblque Endoscope 45. The remander of ths paper s organzed as follows: Secton 2 presents a method for endoscope calbraton. In secton 3, marker desgn and trackng algorthm are dscussed. In secton 4, endoscope mage dstorton correcton method s descrbed n detal. To nvestgate the trackng results, the desgn and smulaton of a camera manpulator are descrbed n the secton 5. Fnall the expermental results are shown and future work s dscussed. 2. Calbraton Method Defnng the pxel-to-world mappng s known as camera calbraton. Once the camera has been calbrated, we can transform the mage pxel coordnates to ther real-

4 world equvalents. Camera calbraton s a crtcal frst step n man applcatons such as dmensonal measurement of mechancal parts, trackng, camera-on-robot confguraton and robot vehcle gudance. Camera calbraton has been nvestgated extensvel b man researchers (e.g., [8], [9]). However, methods for camera calbraton usng rgd endoscope have not been addressed before. In general, endoscope mages have partcular characterstcs. The endoscope vewng angle, tpcall between feld of vew of human ees. Thus, vson s lmted b the narrow endoscope feld of vew. To enlarge the feld of vew of endoscopes, wde-angle lens are emploed n the endoscope. However wde-angle lenses ntrnscall cause barrel dstorton (.e., transformaton of straght lne nto curves). Barrel dstorton cause obects n the mage to be dstorted along radal lnes from the mage center. Ths tpe of dstorton s non-lnear: mage areas farther awa from the center appear sgnfcantl smaller. Ths paper proposes a calbraton method that assumes the mage located wthn the small center area s dstorton free. Tpcal calbraton parameters can be classfed nto two classes: extrnsc parameters and ntrnsc parameters. Ths secton presents methods for determnng ntrnsc parameters, namel, the effectve focal length, the real mage center and the scale factor. The extrnsc parameters, that s, the rgd bod transformaton from the world coordnate sstem to the camera coordnate sstem s dscussed later. ο ο 65 75, as shown n Fgure 3, s narrow compare to the wde Fgure 4 llustrates the basc geometr of the camera model wth ( x,, z) as the world coordnate sstem. In robotc sstem, ths coordnate frame can be placed at the robot base. x,, z ) s the camera coordnate sstem, Pxz (,, ) s the 3D coordnates of a ( c c c z camera o c V ( V, V ) B P( x,, z) or Pc ( xc, c, zc ) f xd o ( V 0, V 0) z c x c d c V x o w (a) x c c o c f o V f V (b) z c Fgure 4. Camera geometr wth perspectve proecton. (c) z c

5 obect P n the world coordnate sstem, Px ( c, c, z c) s the coordnate of pont P n the camera coordnate sstem wth ts z c axs the same as the optcal axs. ( x d, d ) (not shown n Fgure 4) s the mage coordnates of P ( xc, c, zc ) proected nto the mage plane f a perfect pnhole model assumpton s used [10]. However, snce the unt for the coordnate used n the computer, ( V, V ), s the number of pxels n the frame memor, addtonal scale factors need to be specfed that relate the camera coordnate sstem to the computer mage coordnate sstem (usng pxel unts). The focal length f s the dstance between the mage plane center, o, and the optcal center, o c. Here we defne the mage center as the pont where the camera optcal axs passes through the mage plane. In theor, the mage center should be the center of the mage plane. However, n practce, especall n the endoscope mage, the real mage center s dstorted due to the mperfect grnd of endoscope lens. In the followng, frst we propose a smple and fast method to obtan the real mage center. Then, a method for calculatng the calbraton parameters such as the effectve focal length and the scale factor. Fgure 5 shows the calbraton flow chart. ( xc, c, zc ) 3D camera coordnate sstem Step1. Fnd the real mage center Step 2. parameters to be calbrated : focal length Step 3. Parmeters to be calbrated: scale fectors ( V, V ) mage buffer coordante Fgure 5. Three steps n endoscope calbraton. The frst problem that needs to be addressed s the method for calculatng the real mage center. Snce abdomnal cavt s dark, an external lght source s used, see Fgure 3. A fber optc cable transmts lght to the sde of the endoscope. The lght source s evenl dstrbuted through the rod lens. If we pont the endoscope perpendcular to a whte background, the lght source forms a whte crcle n the mage as shown n Fgure 6. (a). Inspred b ths feature, we propose a smple method to obtan the real mage center b pontng the endoscope to a whte background and recordng the mage. The center of the whte crcle n the mage s defned as the real mage center as seen n Fgure 6. (b). Other work, such as Asar et al. [11], consdered the dstorton n endoscope mage and proposed to use straght-lne patterns. Then chose the ntersecton of lnes that reman straght n the endoscope mage as the real mage center. However, ther methods need

6 complex calculatons. The feasblt of the proposed method for determnng the real (a) (b) Fgure 6. Endoscopc mage (a) orgnal gra scale mage, (b) bnar mage (the mage center has been marked). mage center s evaluated later n the expermental results secton. The second parameter to be calbrated s the effectve focal length. As shown n Fgure 4, the transformaton from the 3D camera coordnate ( x c, c, zc ) to the mage coordnate ( V, V ) based on the perspectve proecton wth pnhole camera geometr elds the followng equatons: oo ( 0 0) c f ( V V ) S V V S x = = =, zc zc xc c (1) z ( ( 0) c 0) c V z V V S V S x xc =, c =, f f where oo c s the dstance between the center of the mage buffer to the center of the camera coordnate frame, ( V 0, V 0 ) s the mage center n the pxel unt, S, S x are the scale factors that map ( x d, d ) n the camera coordnate frame to ( V, V ) n the computer mage coordnate frame. When V, V, X and c Z are known parameters the c effectve focal length can be calculated from equaton (1). Also, when f, V, V and x c, c are known parameters, we can calculate the depth nformaton, Z c, of an obect n d x d Fgure 7. The pattern mage for calbraton.

7 the feld of vew. Scale factors S S x, can be obtaned usng the followng formula xd = ( V - V0) Sx, d = ( V -V0) S. (2) In general, manufacturers of CCD cameras suppl the nformaton of center-to-center dstance between adacent sensor elements n the Y drecton (.e., scale factor S ) as a fxed value: S = N fy / d, where N fy s the number of pxels n the Y drecton; d s the dmenson of CCD n the Y drecton. The scale factor S x s an uncertan value due to varous reasons, such as slght hardware tmng msmatch between mage acquston hardware and camera scannng hardware, or the mprecson of the tmng of TV scannng tself. Here we propose a smple and fast method to obtan the relatonshp between S x and S. B capturng an mage of known dmenson (e.g., the square grds, d x and d are the actual center-to-center dstance n mllmeters of adacent squares, see Fgure 7), we can compute ther correspondng, D x and D, n pxel unts n the computer mage coordnate. The followng relatonshp exsts: Dx = Sxdx, D = Sd. Solvng for S x, we get S = S D / D. (3) x x In actual mplementaton, we can measure several squares center-to-center dstances and take the mean value to reduce the possble error n mage processng. 3. Lens Dstorton and Correcton Methods No lens can produce perfect mages. Common mperfectons are aberratons that degrade the qualt or sharpness of the mage and lens dstortons that deterorate the geometrc qualt (or postonal accurac) of the mage. Endoscope mage has a fundamental lens dstorton due to the wde-angle desgn of the endoscope s obectve lens. Wde-angle lenses are used n endoscope because the provde larger vewng felds. However, lens dstorton wll result n erroneous measurement for mage postons n the resultng mages. To elmnate the dstorton effect, correctons should be appled to the measurement of the resultng endoscope mages. Lens dstortons are classfed as ether radal or tangental. Radal dstorton, as ts Fgure 8. Barrel tpe dstorton (The left mage s an undstorted grd pattern, and the rght mage s the same pattern vewed after radal dstorton).

8 name mples, causes mage poston to be dstorted along radal lne from the optcal axs. Radal dstorton ncludes barrel dstorton, pncushon dstorton or the combnaton of these two tpes. Tangental dstorton s due to mperfect centerng of the lens components and other manufacturng defects n a compound lens, and s also called decenterng. That s, the optcal center and the lenses are not strctl collnear. The pxel shft s awa from the optcal center and the new poston les at a new angle locaton as measured from the optcal center. Tangental lens dstortons are generall ver small and are seldom corrected for. Barrel tpe radal dstorton s common n laparoscopc mages, see Fgure 8. Areas further from the center of the feld of vew appear smaller than the reall are. In ths paper, we onl consder the barrel tpe dstorton. We propose a dstorton correcton method that approxmates the dstorton curve b a polnomal and uses the least squares method to fnd the coeffcents for ths polnomal. Wth ths expermentall determned polnomal, we can correct the endoscope mage accordngl. 3.1 Radal Lens Dstorton Model The radal lens dstorton model conssts of two mage planes. The dstorted mage plane s represented b( V, V ), whle the dstorton correcton mage plane s represented b ( V, V ). Ther correspondng centers are represented b( V 0, V 0 ) and ( V 0, V 0 ). The center of the dstorton-correcton mage can be chosen arbtrarl. Unlke the method proposed b [12] that selects these two centers at dfferent locatons, n ths paper, we select both centers the same as the real mage center. The real mage center s defned as the pont where straght lnes reman straght as the pass through ths pont n the dstorted mage plane. Usng a polar coordnate sstem wth ts orgn at the real mage center ( V0, V 0), a pont P n the dstorted mage plane can be represented as 2 2 r = ( V V ) + ( V V ), 0 0 V V (4) 0 θ = arctan( ), V V 0 where magntude, r, s the dstance of P to the center ( V 0, V 0 ). The correspondng pont, P, n the corrected mage plane s P. Pont P can be represent n polar coordnate as ( r, θ ). Because we assume that the dstorton s pure radal, the polar angle s unchanged n the dstorted and corrected mage planes. Hence:θ = θ. The obectve of the dstorton model s to fnd the relatonshp between P and P. As shown n Fgure 9, the relatonshp between r and r s r = r r, (5) where r can be represented as an odd-ordered polnomal seres[13]: r = k r + k r + k r + k r... (6)

9 where k ( = 1,2,3...) are the expanson coeffcents. After obtanng the coeffcents, the new pxel locaton n the corrected mage plane can be calculated as: r 2 4 V = V V = V(1 ) = V(1 k1 k2r k3r...) r (7) r 2 4 V = V V = V(1 ) = V(1 k1 k2r k3r...). r To obtan the polnomal seres for mappng the dstorted mage to the dstorton correcton mage, frst, we need to obtan the coeffcents for the polnomal. 3.2 Coeffcents Estmaton The polnomal coeffcents defne the shape of the curve. The can be calculated b nonlnear regresson analss, such as the least squares method, to obtan the best curve ft to a gven data set (obtaned from experments). A pattern mage (see Fgure 7) was used to obtan the gven data set. Snce we assume that the dstorton s pure radal, the dstorton s crcularl smmetrc. Wthout the loss of generalt, testng squares that le n the horzontal or vertcal of the mage can represent the general dstorton pattern for the whole mage. For a gven data set S = { C1, C2,..., C,... CN}, let there be N columns of testng squares, for each C ( r, r ), r represents the radal dstance between the center of the square from the real mage center, r s the dstorton at a radal dstance r. Consder the polnomal seres of degree 2M + 1: r = k0r + k1r + k2 r kr +..., = 0,... M (8) The devaton of a pont from the above equaton s M 2 1 = 0 F = k r +. (9) o v V r r PV (, V) V x d ( V, V ) 0 0 V θ PV (, V ) d V Fgure 9. Components of Radal Dstorton. The mage pont V, ) s radcall dsplaced b r to the new poston (, ) V. V ( V

10 The least squares problem s then to fnd the values of k, = 0,.., M, so as to mnmze N M [ ( kr + )] (10) = 1 = 0 Hence, k can be calculated b F = 0, for = 01,,..., M. (11) k From equaton (9), M + 1smultaneous equatons are obtaned and represented n the matrx form: AK = Y. (12) where (a) (b) Fgure 10. (a) The orgnal mage, (b) the mage after dstorton correcton (usng forward mappng). A= r for = N = M [ ] ( N+ 1) ( M+ 1), 0,.., 0,...,, T [ 0, 1,..., M ], T = [ 0, 1,..., N ]. K = k k k and Y Equaton (12) can be calculated as T - T K = ( A A) 1 A Y. (13) The matrx K conssts of most probable values for unknowns, k 0, k 1,..., k M. Once the expanson coeffcents are computed, all pxels from the dstorted mage are mapped onto the corrected mage. However, the drecton of mappng s one problem to be consdered. One approach s mappng from the dstorted mage plane to the corrected mage plane usng the formula of r = r + r or r = k 1r + k 2r + k 3r + k 4 r +... (14) where r s the radal dstance measured n the dstorted mage plane. r s the radal dstance measured n the corrected mage plane. However, ths method of mappng wll result n blank pxels n the corrected mage due to the non-lnear expanson of the mage. The orgnal mage has to be mapped nto a new enlarged mage. See Fgure 10 (b),

11 notce there are man blank pxels n the mage, especall n the perpheral areas. Ths problem can be avoded b usng an nverse-mappng method, that s, mappng from the corrected mage plane to the dstorted mage plane. The coeffcent estmaton method descrbed n equaton (7) used the nverse mappng method. In equaton (7), r was measured n the corrected mage plane. Thus, for ever pxel n the corrected mage plane, the correspondng locaton n the dstorted mage s obtaned usng the polnomal (8). The nformaton (e.g., gra level) for that pxel locaton s assgned to the pxel n the corrected mage plane. In the event that the pxel postons calculated usng nverse mappng are non-ntegers, we smpl round them to the closest ntegers. Because the corrected mage enlarged the dstorted mage n barrel tpe radal dstorton, several pxels n the corrected mage ma possbl fnd the same pxel n the orgnal mage. In ths manner, all pxels n the corrected mage plane can fnd ther correspondng pxel values, thus generatng a complete undstorted mage. 4. Marker Desgn As mentoned n the ntroducton secton, researchers have proposed several tpes of trackng methods such as optcal flow (the optc flow feld s the 2D dstrbuton of apparent veloctes that assocated wth the varaton of brghtness patterns on the proecton), feature ponts correspondence and model-based trackng. In our applcaton, due to the dverse desgn of nstruments, t s dffcult to locate the nstrument drectl. To solve ths problem, a black strp marker s desgned. The marker s attached to the tp of nstruments, whch s then dentfed for the trackng task. In general, the trackng task ncludes recognzng markers and calculatng ts relatve poston n the camera coordnate sstem. In real surger, the nstrument ma enter the feld of vew from dfferent postons and at dfferent angles, usng shape analss and pattern matchng would be nfeasble, for there s no fxed shape suted for trackng. Instead, the mage contrast nformaton s used alone for nstrument segmentaton to locate the marker s poston n the mage. Two factors should be consdered n the marker desgn. For one thng, n real-tme mage trackng, the marker should be smple and eas to locate. For the other, the marker s sze should be chosen accordng to the sze of the nstruments. Consderng the above constrans, we desgn the marker as shown n Fgure 11. The strps n Fgure 11 are the desgned markers. The are of the same sze, each wth the P1 P2 P3 2d 2d M d d d Fgure 11. The desgned marker.

12 wdth of d ( M /3 < d < M /2). The dameter of the nstrument s M. Ponts P1, P 2 and P 3 are the centers of each strp proected onto the mage plane. These strps center-tocenter dstance s 2d. Desgnng the marker n ths shape has several advantages. Frst, the shape of the marker s smple and ts contrast wth the nstruments and background s large; thus, t s eas to dentf the marker. Second, the tool s dameter M s used to acqure depth nformaton. Regardless how the nstrument rotates along ts axs (see the dotted lne n Fgure 11), we can alwas fnd two perpheral ponts on the marker that can represent the tool s dameter. Fnall, even f an organ or other nstruments block one or two strps, we can stll get the nstrument postonal nformaton form the remanng strps (at least one strp should be n the endoscope feld of vew). When needed, the trackng task s to move camera so as to poston an nstrument feature, such as markers, at the desred locaton of the endoscope feld of vew and also keep the depth of the chosen nstrument feature as a gven value as the nstrument moves. Consderng Fgure 11, durng the trackng procedure, one of the ponts among P1, P 2, and P 3 s located, the located pont has not been blocked and s the closest to the tp of the nstrument. To determne whch pont s the closet one to the tp, we can compare ther dameter M value n the mage. We assume that n laparosocopc surger, generall the tp of the nstrument s the pont that s the furthest awa from the endosocpe; thus, the smallest M s the one closed to the tp of the endoscope. Based on the change of the chosen pont n V and V drectons n the mage plane and the calbraton parameters descrbed n secton 2, we can calculate the dsplacement n real world unt to whch the endoscope should be moved. From the change of dameter M n the mage plane, the depth nformaton of the marker can also be obtaned usng formula (1). Ths nformaton s utlzed as feedback for manpulatng the endoscope so as to keep the tp of the nstrument at a desred poston n the endoscope feld of vew. 5. Endoscope Holder Desgn In laparosocopc surger, the endoscope s nserted nto the abdomnal cavt through a trocar. Constrans at the pvot pont onl allow four degrees of freedom for Endoscope Holder Pvort Pont θ ρ φ Abdomen Wall γ Laparoscope Fgure DOF constrants mposed b the pvot pont on the endoscope.

13 manpulatng the endoscope, that s, three degree of freedom of angular movement ( θ, φ, γ ) around the ncson pont and one translatonal degree of freedom ρ n or out the ncson pont (see Fgure 12). These sphercal movement constrans should be carefull consdered pror to the desgn of the endoscope holder. Consderng Fgure. 12, we notce that rotaton, γ, s onl used to adust the orentaton of the mage. Generall ths angle wll not change durng the surger. Based on ths observaton, the endoscope holder desgn can be smplfed as 3DOF desgn wth two rotatonal DOF and one translatonal DOF. For robot manpulators, there are two tpes of onts, P: prsmatc (prsmatc onts exhbts sldng or lnear moton) and R: revolute (revolute onts exhbt rotar moton about an axs). The endoscope holder has to have sphercal movements. Among dfferent combnatons of prsmatc and revolute onts, 3DOF RRP produces a sphercal coordnate robot, whch fts the purpose of ths work. Another notable problem s that the desgned endoscope holder should grantee the fxed poston at the pvot pont. Consderng all the constrants stated above, a desgned endoscope holder s llustrated n Fgure 13. Ths desgn satsfes the knematc constrans. To ensure that the endoscope holder wll not change ts poston at the ncson pont, we can adust pont, P, to the same heght as the ncson pont. The world coordnate frame, ( x,, z), s set at the base of the endoscope holder. Generall, the camera s mounted at the end of the endoscope outsde the abdomen. The endoscope transmts the mage as seen at the tp of the endoscope nsde the abdomen. Hence, we can set the camera coordnate frame, x,, z ), at the center of the tp of the endoscope. ( c c c θ ρ θ P φ ϕ ρ 1 z c z c x c x Fgure 13. The desgned scope holder.

14 The trackng task s to obtan the three dsplacement parameters, θ, φ and ρ, based on the mage features. These parameters are used to control the endoscope holder. As shown n Fgure 14, here the desred poston for the marker s chosen as the mage center ( V 0, V 0 ). When trackng, frst, we determne the errors, V and V, between the current marker locaton and the desred locaton, and then transform them to real world coordnates usng the calbraton parameters descrbed n secton 2. Second, the followng formula exsts for small angles: α = d / ρ. Replace d wth V and V, α wth θ and φ, we obtan: θ = V / ρ (14) φ = V / ρ The translaton parameter, ρ, s calculated from the current dameter M n the mage plane usng formula (1). Last, the endoscope can be adusted accordngl to keep the nstrument at a desred locaton under the endoscope feld of vew. V Image Plane x d V ( V, V ) 0 0 ρ α V ( V, V ) d d Fgure 14. The geometrc approach for obtanng three dsplacement parameters. V 6. Expermental Results To evaluate the proposed method, several tpes of experments are conducted. In ths secton, we descrbe the camera calbraton, endoscope mage dstorton-correcton, endoscope holder smulaton and vsual trackng results separatel. An electronc endoscop sstem, comprsng a Karl Storz endoscope[12] s used n capturng the mage. A testng pattern-mage contanng a rectangular arra of black squares of 4 4 mm n sze, separate b 4mm n the horzontal and vertcal drecton (see fgure 7) s used for calbraton and dstorton correcton purpose.

15 The horzontal lne The vertcal lne Fgure 15. Verf the proposed center determnaton method. 6.1 Calbraton Results A dscusson of the calbraton parameters, whch are the real mage center, the scale factors, and the focal length, and how the are obtaned, s found n ths secton. 1) Locate the real mage center. Before the experment, a grd pattern s used to evaluate the feasblt of the proposed center determnaton method. Frst, the real mage center s located b the proposed method. Then the software draws two crossed long lnes wth ther ntersectons at the real mage center usng an overla mage. Thrd, place the pattern mage under the ensoscope feld of vew. The pattern mage should be placed b pckng one square n the pattern mage and algnng ts sdes wth the two marked long lnes. At the same tme, algn the chosen square s center wth the real mage center (see Fgure 15). As we can see n fgure 15, lnes passng through real mage center reman straght n the endoscope mage. Numercal analss also shows that for one row of squares that travel along the horzontal lne, ther centers have the same V coordnates. For one column of squares that travel along the vertcal lne, ther centers have the same V coordnates, whle other columns or rows of squares do not have ths feature. It shows the proposed method s feasble. (a) (b) Fgure 16. (a) Orgnal gra scale mage, (b) Bnar mage, the blob center have been marked.

16 Applng ths method, the real mage center could be obtaned as follows. An mage s recorded b pontng the endoscope perpendcular to a whte background. The captured mage s then processed b bnarzaton usng a threshold. The center of the whte crcle n the bnarzed mage s chosen as the real mage center (see Secton 2, Fgure 6). 2) Obtanng mage coordnates of the calbraton ponts. The centers of squares are chosen as the ponts for calbraton. Because ths work s onl nterested n obtanng the ntrnsc parameters, we onl need to measure the adacent square s center-to-center dstance. Ther values n the world coordnate unt are known. Image coordnates of the calbraton ponts are computed as follows: a) Acqure a gra scale mage (see Fgure 16 (a)). b) Threshold the mage and calculate the center of ever blob (see Fgure 16 (b)). Because the threshold value s not crtcal and the llumnaton sstem s kept stable n our applcaton (It can also be selected automatcall b analss of ntenst hstograms f necessar). 3) Calculatng camera ntrnsc parameters. Ths secton descrbes steps for calculatng camera ntrnsc parameters as well as the calculated ntrnsc parameters of the endoscope used n ths work. a) Scale factors. In ths step, frst we examne the valdt of the assumpton that the dstorton n the center area s neglgble. The closer the endoscope s to the obect, the more dstorted the mage appears. After obtanng the real mage center, the test grd s placed perpendcularl 50mm (the tpcal range for endoscope surger s 40mm-100mm) awa from the endoscope, wth the center of one square at the real mage center. Fgure 16 shows the orgnal gra scale endoscope mage and the mage after beng bnarzed and marked the center of each blob. To reduce the nfluence of lens dstorton, we use the 9 central squares for calculaton. As shown n Table 1, from the coordnates of the center of Table 1 V, V coordnates of the center of 9 squares n the central area (pxel unt) (298, 196) (363, 168) (430, 169) (297, 234) (364, 234) (431, 234) (297, 300) (363, 301) (431, 300) these 9 squares, ther correspondng V or V coordnates are onl at most 1 pxel dfference. Gven the frst column n the table as an example, the coordnates of the three ponts along V drecton are 298, 297, 297, onl one pxel s dfference from each other. Ths dfference s neglgble for our current applcaton. We use mean values of the center-to-center dstance between adacent squares along V and V drectons to calculate the scale factors. Sx S = 65.83! b) Focal length.

17 Consderng the above steps, we note that the scale factors n V and V drectons are almost the same. In ths task, we do not need to calculate the value of focal length n the world coordnate unt, the focal length can be calculated as: ( V -V0 ) Zc f = = = (pxel) Xc 8 3) Calbraton Data The depth nformaton can be obtaned b usng the calculated focal length and equaton (1) as descrbed n Secton 2. The mage of the markers are record at varous dstances from the endoscope (40, 60, 70, 80 and 90 mm). The expermental data are shown n Table 2. Errors n ths depth measurement experment comes from dfferent sources, such as the mage processng error, the accurac of phscall measured dstance, and the actual angle of the laparoscope wth the expermental plane (we assume t as 90 o ). Table 2. Expermental data va the proposed algorthm. No. Actual dstance (mm) Calculated dstance (mm) Errors (mm) Dstorton Correcton Results For the proposed dstorton correcton procedure, the same calbraton chart contanng a rectangular arra of black squares s used. The testng chart s placed perpendcularl to the camera s optcal axs. Fgure 17(a) shows the captured mage. We compute the polnomal coeffcents usng the method proposed n Secton 3. The corrected mage s shown n Fgure 17(b). The mage s (a) (b) Fgure 17. (a) Dstorted mage of testng mage, (b) corrected mage correspondng to (a) usng nverse mappng.

18 magnfed after correcton, and lnes are straghtened especall n the perpheral part. The least total error reduces when the order of the polnomal becomes hgher. Nonetheless, wth the ncrease of the order of polnomal, the computaton tme also ncreases. The varaton n least-square total error was neglgble beond the 3 rd polnomal order (see Table 3), therefore, we pck the polnomal order to be 3 as a compromse between computaton tme and error. Table 3. Varaton n least total error wth respect to order of expanson polnomal. Polnomal order n Least square total error The relatonshp between the radal dstance of the testng ponts (n the experment, we chose squares les n the same row) from the real mage center before and after dstorton correcton s shown n Table 4. Note that the calculated value has a decmal part. Snce the mage buffer accepts nteger pxel locatons, the floatng number s rounded to ts closest nteger. After the dstorton correcton, the radal dstance s more evenl dstrbuted and closer to the actual radal dstance. Table 4. The radal dstance of testng ponts from the real mage center before and after dstorton correcton. Idea Radal Dstance(pxel) Radal Dstance (pxel) (before correcton) Radal Dstance (pxel) (after correcton) (22) (56) (84) (112) (140) The above fgures and expermental results show that the proposed dstorton correcton method elds satsfactor results for vsual and computer analss. 6.3 Endoscope Holder Smulaton Results To vsualze and valdate the endoscope holder desgn. OpenGL was used to smulate the anmated control of the endoscope holder through the data obtaned from the mage trackng process. Usng Vsual C++ splt wndow technques; the smulated endoscope holder and the real endoscope mage captured n real tme are shown smultaneousl n Fgure 18. The left sde of the mage shows the endoscope mage whle the rght sde shows the anmated endocope holder (the small block represents the mark attached to the nstrument). The endoscope holder moves accordng to the postonal nformaton obtaned from the marker that attached n the nstrument ndcated n the left mage. The endoscope holder can be controlled n real tme b movng the tool under the endoscope.

19 Marker poston Fgure 18. Endoscope holder smulaton result. 6.4 Trackng Results To estmate errors n automated nstrument localzaton and trackng wth the proposed method, several measurement methods are appled to evaluate the results. Besdes drect measurement, pcbrd sstem[12] (a poston and orentaton measurement sstem) s also used to obtan postonal data for comparson wth the trackng results. Fgure 19 shows the expermental setup. The endoscope s fxed nto a stand n the vsual trackng experment. The pcbrd comprses a transmtter, a recever (sensor). The pcbrd measures the poston and orentaton of a sensor b transmttng a pulsed DC magnetc feld measured b the sensor. From the measured magnetc feld characterstcs, the chp computes the sensor s poston and orentaton wth respect to the center of the transmtter. In the experment, the sensor s mounted n the upper part of a tool, whle the Camera lght source Endoscope pcbrd Sensor computer screen Instrument pcbrd Transmtter Fgure 19. Expermental setup for the trackng experment

20 r 4 Z drecton b 1 b 2 r 1 b 4 t 1 r 2 t 2 t 4 r 3 t b Y drecton X drecton 4 5 Fgure 20. Comparson of trackng results (ntal b represent the pcbrd trackng results, t represents the proposed trackng results, r represents the ruler measured real postons). maker s attached to the tp of the tool. Snce the relatve poston of the sensor and the marker s fxed, the marker s poston can be derved from the sensor s poston. Thus, the marker s 3D poston obtaned from the pcbrd and from the proposed mage trackng method can be compared. We defne a world coordnate frame at the center of the transmtter. The camera coordnate frame s defned at the tp of the endoscope. Four ponts wth dfferent locaton are gven for the experment. We place the tool at the gven locaton and let the center of the marker (select one strp of the marker) overlap wth the gven ponts. In ths experment, we capture ten frames n each locaton. The mean trackng error s defned as the average dfference over a whole sequence between the actual measured poston and the poston obtaned from the proposed trackng methods. A smlar approach s appled to the pcbrd poston and orentaton measurement sstem. The results are shown n Fgure 20 and Table 5. The error s measured n terms of cm and n the x, and z drectons. From Fgure 20, the vsual trackng results are better than the pcbrd poston and orentaton measurement sstem. One reason s related to the restrctons n usng the pcbrd. The pcbrd s supposed to be used n a non-metallc envronment. However the operatng room envronment cannot guarantee ths restrcton. The errors for both trackng methods are affected b mprecse manual placement of the marker at the known poston. Table 5. Average Error n Instrument Trackng Sequence No. Mean trackng error (cm) (10 frames) x drecton drecton z drecton pcbrd Image Trackng pcbrd Image Trackng pcbrd Image Trackng

21 7. Concludng Remarks In ths paper, some novel deas of usng vsual trackng to realze automated nstrument localzaton and endoscope maneuverng n robot-asssted laparoscopc surger are presented. Implementaton methods are descrbed n detal. Good expermental results demonstrate the feasblt of the proposed methods. Some more experments can be conducted n future work n the vsual trackng part, such as, testng the trackng accurac when the tools enter the endoscope feld of vew from dfferent angles. In our proposed method, we use the ntenst nformaton of the mage for the marker localzaton, the specular reflecton due to lghts proected nto the shnng obects does not affect the trackng results. Ths s because n gra scale mages, the ntenst dfference between the shnng obects and the marker s even larger than surroundngs, whch s a desred feature for locatng the marker. The general goal of ths work s to further mprove the exstng laparoscopc surger sstem, to buld a human machne nterface for the surgeon to automatcall manpulate the endoscope. The sstem can be appled to surgcal tranng smulators and telepresence (telepresence sstems are used n medcne to manpulate equpment at a remote ste. The surgeon has the sense of actuall beng at the ste performng the procedure). Our future research goal s to utlze ths technque together wth other sensors, such as laser sensors to reconstruct the surface of the surgcal ste, so to overcome the lack of 3D depth cue of the current laparoscopc surger sstems. The research wll beneft the development of the laparoscopc surger sstem n the long run. References: [1] L.Mettler, M. Ibrahm and W. Jonat, One Year of Experence Workng wth the Ad of a Robotc Assstant the Voce-controlled optc holder AESOP n Gnaecologcal Endoscopc Surger, Human Reproducton, Vol. 13, No. 10, 1998, pp [2] Davd Prewer, Les Ktchen. A Smple Fast Edge-Based Vsual Tracker. Techncal Report 97/20, pp [3] A. Azarbaean and A. Pentland, Real-tme self-calbratng stereo person trackng usng 3-D shape estmaton from blob features, Proceedngs of Internatonal Conference on Pattern Recognton, Aug. 1996, pp [4] Redd, S. and G. Lozou, Analss of camera behavor durng trackng, IEEE Transactons on Pattern Analss and Machne Intellgence, volume 17 (1995), number 8, pp [5] M. E. Leventon, Baesan estmaton of 3-d human moton from an mage sequence Technque report. Jul [6] Y. Wang, D. R. Uecker and Y. Wang, Choreographed Scope Maneuverng n Robotcall-Asssted Laparoscop wth Actve Vson Gudance, IEEE Workshop on Applcaton of Computer Vson, Saratoga, FL, Dec. 1996, pp

22 [7] G. We, K. Arbter and G. Hrznger, Real Tme Vsual Servong for Laparoscopc Surger, IEEE n Engneerng and Bolog, Jan. 1997, pp [8] C. Chen, S. Sttt and Y.F. Zheng, Robotc Ee-n-Hand Calbraton b Calbratng Optcal Axs and Target Pattern, Journal of Intellgent and Robotc Sstem, 12, 1995, pp [9] R. Y. Tsa, A Versatle Camera Calbraton Technque for Hgh- Accurac 3D Machne Vson Metrolog Usng Off-the-Sheld TV Cameras and Lenses, IEEE Journal of Robotcs and Automaton, Vol. RA-3, No. 4. Aug. 1987, pp [10] James D. Fole, Andres van Dam, Steven K. Fener, John F. Hughes, Computer Graphcs: Prncples and Practce, Addson-Wesle [11] K.V. Asar, S. Kumar and D. Rashakrshnan, Technque of Dstorton Correcton n Endoscopc mages Usng a Polnomal Expanson, Medcal &Bologcal Engneerng & Computng, vol.37, 1999, pp [12] W. E. Smth, N. Vakl and S. A. Masln, Correcton of Dstorton n Endoscope Images, IEEE Transactons on Medcal Imagng, Vol. 11. No , pp [13] P. Wolf, Elements of Photogrammetr, Mcgraw-Hll Inc [14] KARL STORZ Endoscop Canada Ltd, [15] Ascenson Technolog Corporaton,