Laser-Pointing Endoscope System for Intra-Operative

Transcription

1 Proceedings of the 2001 IEEE International Conference on Robotics 8 Automation Seoul, Korea. May 21-26, 2001 Laser-Pointing Endoscope System for Intra-Operative 3D Geometric Registration Mitsuhiro Hayashibe Yoshihiko Nakamura ( [hayasibe, nakamura] eynl. t. U- tokyo. ac. j p) Department of Mechano-Informatics University of Tokyo 7-3-1, Hongo, Bunkyoku, Tokyo Japan Abstract Precise measurements of geometry should accompany robotic equipments in operating rooms if their advanhges are further pursued. For deforming organs including a liver, intraoperative geometric measurements play an essential role in computer surgery in addition, to pre-operative geometric information from CT, MRI and so on. We developed a laser-pointing endoscope using an optical galvano scanner and a 955fps hi,qh-speed camera. The laser-pointing endoscope system acquires and visualizes the shape of the area of interest in a flash of time. Applications of the system also include the touch screen interface for non-master-slave operation of surgical robots, where the 30 coordinates of the touched point on screen are measured by the system and guide a robot. Results of in-vivo experiments on a liver of pig verify the eflectiveness of the proposed system. Key Words: medical robotics, m,inimally invasive surgery, endoscope, laser scanner, 30 geometry 1 Introduction Endoscopic surgery forces surgeons to operate with mental tcnsiori under the mechanical and visual constraints. Thc visual difficulties are due to narrow sight of endoscopcs and the lack of depth perception. Minimally invasive surgery would be technologically improved if surgeons are provided with the 3D shape of internal geometry and the 3D coordinates of the point of interest, in an intuitive manner. The related and significant as well would be the issue of human interface. As minimally invasive surgery allows more and more technology involvement[l], tighter but more natural relationship between machines and surgeons are required. Thc Zeus of Computer Motion Inc.[2, 8, 91 and the (la Vinci of Intuitive Surgical Inc.[3], for instance, choose the master-slave configuration, where the human interface is still limited to direct manipulation. In this paper, we develop the laser-pointing endoscope system. We fabricate a prototype using a laser light source, a 2D optical galvano scanner, and a 955fps high-speed camera. This device allows to acquire the intraoperative 3D geometric information in a flash of time. We also propose a touch screen interface so that surgeons can intuitively indicate 3D points of interest on the 2D screen. As an application of the interface, we develope a non-master-slave manipulation system for the surgical robot. Results of in-vivo experiments for a liver of pig verify the effectiveness of the proposed. 2 The Laser-Pointing Endoscope The prototype was developed as shown in Fig.1 employing a laser light source, a 2D galvano scanner, two endoscopic optics, two cameras of different standards and a LCD monitor with touch screen interface. Being controlled by the mirrors of galvano scanner, a laser spot is projected inside the patient s body through an endoscopic optics. We used a closedloop galvano scanner (General Scanning Inc.), which responds up to 1kHz. The laser spot is captured by a 955fps high-speed camera (DALSA Inc.; 256x256 pixels, 256 gray scale) and an image capture/processing board (Viper-Digital; CORECO Inc.). The image of high speed camera is not suitable s monitoring. Information from high speed camera is used only for 3D geometric information of organs. Using a beamsplitting prism, color images of the same scope are captured by an NTSC CCD camera and presented to the surgeon as illustrated in Fig.2. The laser and camera coordinate systems are identified by OPTOTRAK (Northern Digital Inc.) attached to the devices as Fig.4. The 3D coordinates of reference points are reconstructed based on the triangulation between the high-speed camera image and the mirror angles of galvano scanner. Although the image captured through the endoscope is distorted by the relayed lens inside[lo], it is corrected and used to calculate the 3D position /01/$ IEEE 1543

2 Fig. 2: Laser Pointing Endoscope: system configuration 3 Calibrations and Measurements 3.1 Camera Calibration We adopted the parameterization and calculation method of the literature[6]. The relation between the coordinates the point P in the laser frame p and that in the camera frame p is represented as follows: centroid, and s,, sy are the camera scale factors. From Eqs.(4)-(5), we obtain where c+=zc[ H v 3 (6) T.JI = [ sxf Syf CX cy ] (8) where lp= [ ZL YL 75 1 (2) p= [ 2, Yc ZC 1 (3) p is transformed to the 2D coordinates in the image plane by perspective transformation as follows: yv=e f zc f zc -- xu - xc (4) where [ x, yv IT denotes coordinates point P in the image plane, and f is the focal distance of the camera. The transformation from the pixel unit to the unit of the camera coordinates is where [ H V 1 S,X, = cz - H, syyc = cy - V (5) P on the image, [ c, cy IT denotes the pi.xel coordinates of point implies that of the image Because + contains the unknown camera paremeters, they are determined by the camera calibration. Redundant numbers of reference points are measured to obtain the relation between the pixel coordinates in the image and the camera coordinates. The least square computation calculates $7 using pseudo-inverse matrix as follows: 11, =z,c [ H V 1 (9) 3.2 Laser and Endoscope Calibration A laser beam is projected on the internal organ through the endoscope. The end of the endoscope is inserted into the abdominal cavity and laser beam is scanned by the two mirrors of galvano scanner at the other end as illustrated in Fig.3. We made a calibration of the endoscopic optics and determined the relation between input angle and output angle. 1544

3 J YI Fig. 3: input 4 Fig. 1: Laser Pointing Endoscope: the develope1 3 prototype Input angle 4 and output are written as 4 = [ #)yaw 4)pztch 1' = apztch 1' = [ arctan(yl/zl) arctan(zi/zl) ' I (11) Assuming that the relation between $ is a 4yaw + b (') 'I'prtch = c $pitch + d (13) Again using the least square estimation, the parameters of transformation become?r = Lif@ (14) Fig. 4: Laser Frame and Camera Frame Fig.4. The 3D coordinates of the laser mark are obtained as those of the intersection between the line of laser beam LlaseT and the perspective line LcameTa. LlaseT and LcameTa are described as follows: 'p = U [ 1 tan (a$yaw + b) tan ( c $ + ~ d) ~ ]fi7) ~ where L= [ 4yaw syf v-c, (18) 0 4ppiLch Using Eqs. (1), (17), and (18), we can simultaneousn=[ a b c d]'i' (16) ly solve unknowns of lp, cp, and parametric variable U and measure of the 3D coordinates of laser spot D Measurements Triangulation between the laser scanning endoscope and the endoscopic camera (955fps) provides the 3D coordinates of the laser spot as illustrated in 1545

4 4 Experiment 4.1 3D Pointing Interface In the current endoscopic surgery, surgeons have to mentally extract necessary inj'ormation from the image of endoscope. In other words; they must judge the distance and the size of object using their sense. The surgeons would feel less tired if they were released from the mental load. The 3D pointing interface works as follows: (1) the surgeon toughes the screen of endoscopic image at a point of interest. (2) the laser marker is controlled using the galvano scanner so that its position in the endoscopic image converges to the point of interest. (3) Once converged, the 3D coordinates in the camera frame of the point of reference are recovered. We tested the interface and measurements in in-vivo experiments on the liver of a pig. This system could provide the surgeon with the 3D coordinates of the point of interest in real-time. 'The response time from the touch on screen to the output of 3D coordinates was approximately 0.5 second. We will share the trace data that shows how the laser spot were controlled and converged to the target point on the liver. Figure 5 shows the trace of 3D coordinates while the laser marker converges to the point of interest on the liver of pig. This interface could be used to locally illuminate the interest area as shown in Fig.6. It will be useful for intraoperative communication tool under teleoperative environment. tracking(955fps) and laser coni,rol using differential vector in the image. In every frame, differential vector between present laser point and destination is calculated. And laser beam is projected to the direction of the differential vector according to its size until the difference has turned into subthreshold as illustrated in Fig.7. I Fig. 6: Marking on the liver /I Mestination Fig. 7: Convergence in endoscopic image E16 3 hi r' 4.2 3D Geometric Registration Figure 8 shows the scanned surface of 50yen coin by Laser-Pointing Endoscope. The hole on 50yen coin could be measured and the error is within 1%. The scanned 3D data is automatically redescribed in Virtual Reality Modeling Language 2.0 by the our developed program. The shape of liver is easily percieved by www browser with VRML plug-in software. The surface is composed from numerous triangle patches. Fig. 5: The 3D trace of laser mark on the liver The touch screen interface assists a surgeon to intuitively indicate the point of interest on the 2D monitor screen (NTSC). The galvano scanner actively controls the laser spot inside the patient to locate it in the 2D monitor screen (NTSC) at the indicated point by the surgeon. And this system is realized by high speed Fig. 8: Scanned surface of 50yen coin 1546

5 In order to investigate the optical conditions, we scanned tlic 3D shape of pig s liver in laparotomy. Even with the wet and shinny surface condition, it was possible to obtain the 3D positional data using a semiconductor laser light source. The laser power was 15mW. We scanned the area of 20cm square and obtained the 4000 points data. Sampling time took 1.2ms for each point, and the total measuring time was approximakly 5.0 seconds. Fig.9 show the reconstructed 3D VRML image of the scanned liver in in-vivo experiment. Fig. 10: The device settings in laparoscopic surgery Fig. 9: 3D VRML image of liver in laparotomy The image of Fig.9 was obtained after the abdomen was opened. The primal purpose of this research is to obtain 3D geometry under the laparoscopic surgery. We made an iri-vivo experiment under the laparoscopy to obtain the intraoperative 3D geometry as shown in Fig.10. We scanned the area of 8cm square and obtained the 400 points data. Sampling time took 1.2ms for cach point including the calculation time for 3D position, and the total measuring time was 0.5 seconds. The intraoperative 3D geometry of the liver surface could be quickly obtained as Fig.11. The obtained information could be used; for example, for the local surgeon to provide the robotic equipment; controller with a reference point, or for the remote mentoring doctors to numerically measure the location, length, and/or area of interest inside the patient s body. 4.3 Wide Area Registration Due to the narrow sight of the endoscope, the area which is able to be observed is limited for the surgeon. Therefore, it is easy to imagine the possibility of collisions between the organ and forceps of surgical robot in the out of view from the endoscope. To overcome the narrow view, 3D geometry of the region in the out of view is registered into the virtual space. We made an additioiial registration of the data sheet obtained by the scanning. The data sheet contains 400 referent points and is captured within 0.5 second. We made a measurement of the cylinder. The data sheet was Fig. 11: 3D VRML image of liver in laparoscopy added into the virtual space after the position and direction of endoscope is changed to get the different view. Fig.12 shows the three data sheet and the right of it is the side view of them. 4.4 Non-Master-Slave Operation The present interface of surgical robot is limited to master-slave configuration. In the robotic telesurgery using master-slave, the large amount of image data must be transmitted through general network. It brings the difficulties of time delay. As the application of 3D pointing interface, the 3D position of destination for the surgical robot can be obtained from the 2D input on the touch screen.the surgeons have been able to guide the surgical robot to the place where it should approach at the end by the intuitive touch input on the endoscopic image. As shown in Fig.13, The forceps attached to surgical robot could be guided to the object by non-master-slave operation. AESOP (Computer Motion Inc.) is adopted as the surgical robot to have the forceps. 1547

6 References Fig. 12: Registration of out-of-view area [l] Marc 0. Schurr. Robotic Deviccs for Advanced Endoscopic Surgical Proccdures. Journal of thc Robotics Society of Japan CYOL.18 No.1 pp.16-19, 2000 D [a] Computer Motion Inc.. Internet Home Page. [3] Gary S.Guthart. The Intuitive Telesurgery System. Proceedings of the 1E:EE International Conference on Robotics and Automation, pp , [4] S.Baba, H.Kawakami, Y.Nakamura. Laser pointing endoscope for Minimally Invasive Surgery. Proceedings of Robomech 99, 1P ,1999. [5] Y.Nakamura, M.Hayashibe. Laser-Pointing Endoscope System for Natural 3D Interface between Robotic Equipments and Surgeons. Proceedings of Medicine Meets Virtual Reality 2001, pp , [6] H.Zuang, K.Wang, and ZSRoth. Simultaneous Calibration of a Robot and a Hand-Mounded Camera. IEEE Trans. on Robotics and Automation, vol.11, no.5, pp , Fig. 13: The guided forceps by non-master-slave operat,ion 5 Conclusion We developed the laser-pointing endoscope system. The intuitive iriterface to be an intra operative support for surgeons was designed and fabricated using a touch screen and a high-speed camera. The invivo measurement of the pig s liver was carried out. The intraoperativc 3D geometric registration was realized under laparoscopic surgery. Preliminary results of in-vivo experiments verified the functionality and showed the performance. We proposed and realized non-master-slave operation for the surgical robot. Acknowledgements This work was supported in part through Development of Surgical Robotic Systems (PI: Prof. MD. T.Tsuji) under the Research for the Future Program, the Japan Society for the Promotion of Science; and through Telecommunications Advanccment Organization of Japan (PI: Prof. M.M-itsuishi). And I appreciate H.Shimizu for the measurement of device coordinates using OPTOTRAK. [7] N.J.Soper. Essentials of laparoscopy. Quality Medical Pii.blishin.g, [8] H.Reichenspurner, R.I>amiano, M.Mack, D.Boehm, B.Meiser, R.Elgass, B.Reichart. Use of the voice-controlled and computer-assisted surgical system Zeus for endoscopic coronary artery bypass grafting. J Thorac Curdiovusc Surg, Vol. 118, NO. 1: pp , [9] Y.F.Wang D.R.Uecker, C.Lee and Y.Wang. A speech-directed multi-modal man-machine interface for robotically enhanced surgery. Proceedings of the First International Symposium on. Medical Robotics and Com.puter Assisted Surgery, pp , [lo] H.Haneishi, Y.Yagihashi, Y.Miyake. A new method for distortion correction of electronic endoscope images. IEEE Trans. Med. Imag., Vol. 14, pp , September