Optimal Planning of Robotically Assisted Heart Surgery: Transfer Precision in the Operating Room

Transcription

1 Optimal Planning of Robotically Assisted Heart Surgery: Transfer Precision in the Operating Room Ève Coste-Manière 1, Louaï Adhami 1, Fabien Mourgues 1, Olivier Bantiche 1, David Le 2, David Hunt 2, Nick Swarup 2, Ken Salisbury 2, and Gary Guthart 2 1 INRIA Sophia-Antipolis, ChIR surgical robotics, 2 Intuitive Surgical Inc. Abstract. The aim of this work is to quantify the errors introduced at different levels of applying results planned using a computer integrated system (CIS) in the operating room, and make use of these errors to adapt the transfer and rethink the planning. In particular, the registration between preoperative imaging and intraoperative patient model, as well as between the patient and the robot are addressed. Two different registration methods are used and their accuracy compared. Moreover, augmented reality trials are conducted to assess the difficulty of adapting preoperative data to intraoperative models in order to deliver useful information to the surgeon during the intervention. The experimental work in this paper is conducted on a dog for a coronary bypass intervention using the Da Vinci TM surgical system. 1 Theoretical results The benefits of robotic assistance have become apparent in many surgical specialties, including cardiac surgery; however, their potential is still far from being fully exploited in the operating room. The main reasons for this dissimilarity between use and potential stem from the relatively recent use of robots in surgery, which implies new settings, new protocols, new methods for performing the intervention, etc.. One of the best immediate solutions to remedy this mismatch is through a more efficient preparation of the intervention, where the robotic system, the patient and the needs of the intervention are taken into account. Unfortunately, this will also inevitably be coupled with the difficulty of applying the planned results in the operating room, which could invalidate any claimed optimality. As a consequence, being able to measure the errors produced during the transfer enables us to guarantee a level of security on the execution of the planned results. The work presented in this paper focuses on transferring planned results using a computer integrated approach for robotically assisted minimally-invasive surgery (RMIS). A quick overview of the approach is shown in figure 1. This paper concentrates on transferring the planned results into the operating theater. The transfer is a delicate operation for not all factors can be con-

2 2 Ève Coste-Manière et al. Perception Port Placement Robot Positioning Verification & Validation Simulation Transfer Execution / Augmented Reality Analysis Integration: STARS Gather information about the patient, the robot and the environment. Determine the best incision sites (ports) based on intervention requirements, patient anatomy and tools specs. Determine the best relative position of the robot, the patient and the operating theater. Automatically validate planning results under more realistic operating conditions and verify the operating protocol and the corresponding robot logic flow. Rehearse the intervention using patient and robot data in simulated operating conditions. Transfer the results to the operating room through proper registration and positioning. Execute planned results with visual/computational assistance, monitor & predict possible complications. Store the settings and history of the intervention for archiving and subsequent analysis. All the components of this architecture are integrated into a modular single interfaced system: Simulation and Transfer Architecture for Robotic Surgery. Fig. 1. General CIS approach, a more detailed description is found in [2]. trolled, possibly leading to the loss of any claimed optimality or even utility of the planned solutions. In an RMIS procedure, the transfer consists of the following intraoperative steps: register preoperative models to the patient, register the robot to the patient, register the robot to the patient, locate the planned ports on the patient, and give the robot the planned posture. Once the results have been transfered, the intervention is monitored for any predicted or unpredicted complications, and its success is assessed. The effect of the transfer can then be deduced by analyzing the observed complications and comparing expected vs. obtained success rate of the intervention (e.g. overall visibility, dexterity, etc..). 1.1 Registration The registration step is twofold: register the preoperative images (on which the planning results have been obtained) to the patient in intraoperative position, then register the robot to the patient. In the following a fiducial based and a projector based method are presented to carry out the registration. Fiducial based registration Fiducial based registration is a simple and efficient registration technique; however, it necessitates the segmentation of fiducial markers in the preoperative data, as well as their localization intraoperatively. By definition, fiducials markers are usually very distinct in

3 Transfer Precision in the Operating Room 3 preoperative imaging and their delineation can easily be achieved as depicted in figure 3. Moreover, the configuration of the markers should minimize any potential variations between their preoperative and intraoperative positions; e.g., privileging stiff placement such as on the bony structures. Once fiducial points have been obtained in both preoperative images and intraoperative conditions (see figure 3), a rigid transformation is sought in a least-square sense relating the two coordinate frames: the patient coordinate, which is that of the preoperative images and the remote coordinate frame, which is that of the robot. Looking for a rigid transformation assumes the following: The position of the segmented centroids of the fiducials are accurate (e.g. no shift in the z axis in a CT scan). The fiducials did not move between preoperative acquisition and intraoperative measurements. The first assumption depends on the equipment used and is usually easy to meet. On the other hand, the second assumption requires that the fiducials be attached to rigid structures on the patient, which is not always possible, especially when the fiducial markers are glued on the skin. As a consequence, special care should be taken when choosing the location of the fiducials. Surface reconstruction based registration The goal of this method is to register the skin surface segmented from the scan with a 3D reconstruction of the patient in the OR. The reconstruction system is based on a novel structured light method which requires a single image of the scene illuminated by the projector. This results in a small hardware setup (i.e the projector and digital camera mounted on a tripod, both controlled by a laptop computer), and a simple protocol since no respiration gating is needed. Details of the method can be found in [4]). Briefly, we use passive correlation based stereoscopic methods, with the advantage that we can choose one image of the stereoscopic pair (i.e. the projected image) so that the solution to the correspondence problem is both easy to compute and unique since the projected image is a random gray-level pattern. For our experiments, we chose a calibration method that takes as input several views of a single planar grid (a checkerboard) seen at various orientations, as described in [8]. In order to calibrate both the camera and the projector with the same set of images, the random gray-level pattern is projected on the calibration grid throughout the acquisition sequence. An estimation of the homographic relationships between the original pattern and the pattern seen on the grid pictures allows to calculate the positions of grid corners in the projector coordinate frame and thus its calibration. Once the setup is calibrated, the reconstruction is a quite straightforward step, since we use standard correlation based stereo algorithms to solve the correspondence problem. Rectification is done using the camera and projector

4 4 Ève Coste-Manière et al. calibration data, so that corresponding points lie on the same horizontal scanline in both rectified images (for details see section 6.3 of [6]). The disparity map is obtained by matching pattern and images points, using zero-mean normalized cross correlation. The registration of the reconstructed skin surface with the one segmented in the scan is done using an Iterative Closest Point (ICP) algorithm (see [9]). 1.2 Port placement An optimization algorithm is used to determine the best entry points on the segmented anatomy of the dog. The algorithm is based on a semantic description of the intervention, where the main syntactic elements are target points inserted by the surgeon on the anatomical model of the patient to represent the position, orientation and amplitude of the surgical sites. The semantics of the intervention are translated into mathematical criteria that are subsequently used in an exhaustive optimization to yield the most advantageous ports combination for the patient, intervention and robotic tools under consideration. The criteria and their mathematical formulation are detailed in [3]. The position of the ports computed during the planning process are transferred on the patient s skin either using the robot as an interactive pointer, or through a direct video projection. Interactive guidance Once the robot is registered to the patient, the planned ports are expressed in the robot coordinate frame and used to guide the user who will manipulate the end effector as a pointing device, as shown in figure 6. The precision of this approach is limited by that of the registration, as well as the ability of the operator to move the pointing device (the arm end effector). Moreover, the ports should be in an area where the result of the registration remains valid. In other words, the placement of the fiducial markers should take into account a desirable proximity with the potential position of the ports. Direct projection Since the patient has been registered to the scan using the stereo setup, it is quite simple to project any given point chosen from the scan directly on the skin. The projection matrix of the projector, associated with the transformation matrix computed previously gives a direct relationship between the coordinates of the ports in the scan frame and in the projector image frame. 1.3 Robot positioning Planning results are not solely composed of the placement of ports, they also include an optimal position of the robotic system, which is computed in a

5 Transfer Precision in the Operating Room 5 way that maximizes the separation between the robotic arms and with surrounding obstacles, while at the same time insuring fixed spatial constraints (position of the ports) on the arms (see [1]). The positioning concerns passive degrees of freedom (dofs) of the robot: in the case of the Da Vinci TM robot this is the position of the base and the set-up joints, which carry the active (tele-operated) parts of the arms of the robot. Therefore, to reproduce the computed position of the robot in operating conditions, the base of the robot has to be correctly placed with respect to the patient, and its set-up joints should be fixed at the planned values. Alternatively, if not all of the passive dofs can be controlled (e.g. impossibility to move the base of the robot), then the optimization of the positioning is not carried out for the latter, rather they are considered as additional constraints on the system. 1.4 Augmented reality The possibilities offered by augmented reality are boundless, especially in minimally invasive surgery, where reduced visibility is a major concern. For example in TECAB interventions, a view of the operating field (3D with the Da Vinci TM system) enhanced with a 3D model of the coronary tree built from angiography acquisitions would make the work easier for the surgeon (see [5]). This non obvious registration problem between the preoperative model and the endoscopic view can be addressed by: using the result of the external registration of the patient with the robot and the preoperative models (see section 1.1 for details) as an initialization of the superimposition, using the displacement of the endoscope tip deduced from the robot articular values, refining the superimposition with intra-images measurements to deal with the displacement (see section 2.2), deformation and motion of the concerned organs. We propose to interactively solve the registration problem by combining the measures based on automatically extracted reliable landmarks and the indications given by the surgeon. The use of the robot kinematics and the superimposition in the endoscopic images assume that the motorized stereoscopic endoscope is precisely calibrated (see [7] for details on the developed method);i.e., the following should be computed: the optical parameters of the two cameras, the rigid transform between them, and the transformation between the camera frames and the endoscope tip. 2 Experimental validation and results The experiments concern the TECAB intervention, which consists of grafting on a damaged coronary artery another artery (or alternatively using a vein

6 6 Ève Coste-Manière et al. between the two arteries) that would be used as a bypass to irrigate the affected portion of the heart. The entire intervention is performed through incisions in the chest wall, where carbon dioxide is insufflated and the left lung is collapsed, thus enabling the movement of the instruments. The Da Vinci TM surgical system, operated in a special API mode that enables precise readings of the state of the robot and the endoscopic images, is used to operate a dog. 2.1 Preoperative processing & planning Two sets of CT scans where acquired and processed to obtain surface models of the heart of the dog, the bones, LIMA (artery used for the bypass) and the skin, as shown in figure 2, on which the surgeon modeled the intervention through target cones on the heart and on the LIMA. The planning is performed as described in [3] and [1], to obtain optimal placement of the ports and the corresponding collision free position of the robot (figure 2), achieving a minimum separation of 37.8 mm between the robotic arms. Fig. 2. Planning the intervention: Dog anatomy and optimal port placement (left), optimal robot position (right). 2.2 Registration Twelve fiducial markers were manually identified on the CT scan and pointed by one of the robotic arms after the dog had been positioned on the operating table, as shown in figure 3. The obtained RMS error was 19 mm, which is considerably higher than previously observed on plastic phantoms errors, which were typically lower than one centimeter. Looking back at the assumptions behind using a one step rigid transformation and at the measurement process, it is unlikely that the CT scan reconstruction caused the above error, due to the quality of the latter. The same applies to the precision of the robot, which was calibrated before the experiment. Indeed, this step registers the robot directly to the CT scan model, thus implicitly neglecting any non-rigid

7 Transfer Precision in the Operating Room 7 transformation between the scan and the intraoperative dog position. This relatively high error is the counterpart of the simplicity and non-invasiveness of this marker based registration. Fig. 3. Fiducial markers are easily distinguished in preoperative CT images (left). Pointing fiducial markers with the robot tool tip (middle and right). A reconstruction of a skin patch of the dog has been done in order to test the second registration method. It was then registered to the scan with an ICP algorithm, as shown in figure 4. To verify the registration, it had been planned to reproject the fiducials segmented from the scan. Unfortunately, due to constraints in the availability of the lab, the stereo calibration had to be done during the intervention, consuming most of the time granted to this experiment. As a consequence, the reprojection could not be done directly on the dog, but was done instead on the picture from the camera (the markers coordinates were mapped in the camera frame instead of the projector frame). The result is shown in figure 5. Fig. 4. (left) stereo reconstructed patch of skin; (middle) patch registered with scan segmented skin; (right) skin segmented from the scan. Error measurements To better discriminate the different sources of error, a new registration was performed after the insufflation, which introduced a deformation of the rib cage on which the fiducials are distributed. The obtained RMS error was of 20.1 mm, which is comparable to the original error before the insufflation, insinuating that the deformation is within the precision of the registration. On the other hand, the surface based registration gave a RMS error of 10.7 mm. The error was obtained by measuring the distances between cor-

8 8 Ève Coste-Manière et al. responding fiducials in the scan and in the reconstructed patch after ICP registration. This is to be compared with an RMS error of 6.7 mm, corresponding to the transformation that best matches these markers. This latter error is due to geometric distortion between the scan model and the stereo one (deformation due to breathing, position, calibration and reconstruction error). Finally, before the end of the intervention, a reference needle inserted in the thorax of the dog and pointed by the instrument was used to investigate the span of the validity of the registration, as shown in figure 5. An error of 3.2 cm was measured, which is not explained by the registration error. As a consequence, an alternative solution should be sought to correct this shift, such as intraoperative tracking through visual clues (see section 1.4). Fig. 5. (left) Virtual needle used to measure tool tip error. (right) synthetic projection of fiducials. 2.3 Port placement As shown in figure 6, an interactive interface is used to continuously display the error vector between the position of a desired port and one of the arms of the robot. The surgeon uses this information to move the tool tip of the arm to the incision site. The method was judged (clinically) satisfactory in terms of both efficiency and precision. As explained in section 2.2, the locations of ports could not be reprojected intraoperatively, and thus no error measurement could be done directly on the dog. However a synthetic reprojection, which by construction should be very close to what the actual reprojection would have been, is shown in figure 6 (right). 2.4 Robot positioning The robot was positioned to match the stationary remote centers of the arms with the ports, and to avoid any singular or colliding configuration, as shown in figure 2. The minimum separation obtained was of 38.7 mm between the

9 Transfer Precision in the Operating Room 9 Fig. 6. (left) Using the robot as an interactive pointing device for the planned ports: A radar like X-Y and depth indication of the desired port with additional error information is constantly displayed. (right) Synthetic projection of ports. robotic arms. This optimization concerned the robot setup joints and the position of base of the Da Vinci TM system with respect to the patient (total of 9 dofs), and took 132 seconds. Subsequently, and in the absence of any absolute positioning system, the base was moved in a position that matched as much as possible the computed optimal position, after which a new registration was performed and a another optimization of the position of the robot was performed, this time without the position of the base. The resulting position is shown in figure 7, where there was no collision free solution. Indeed, an automatic validation, where the expected movements of the robot are reproduced after the set-up joints have been fixed, showed that a persistent collision occurs with the lower limbs of the dog (maximum of 11 mm), that no collision occurs between the arms, and that a small collision occurs between one of the arms and the operating table (3.2 mm). The conflict with the lower limbs of the dog is not alarming, since mild pressure (1.1 cm) can easily be tolerated. However, a collision with the operating table would lead to a blocking situation. An easy workaround would have been to allow the rotation of the dog in the pose planning, which would have given more flexibility to the relative positioning between the robot and the dog, thus avoiding the colliding state. Nevertheless, it was decided to keep the current position in order to confirm the predicted complications experimentally. Indeed, and despite the high registration error, both collision were observed at the expected sights. Moreover, an unpredicted collision between the arms also occurred, but this was due to the fact that the surgeon was exploring the pericardium, a situation which had not been modeled in the planning phase. 2.5 Augmented reality The endoscope was calibrated on data set of about ten pairs of images (as illustrated on figure 8). Using the pose of the endoscope with respect to the grid, the reprojection error was about 0.5 pixels. After estimation of the transformation between the cameras and the endoscope tip, the reprojection

10 10 Ève Coste-Manière et al. Fig. 7. Final ports and positioning of the robot at the beginning of the intervention. error was about 4.5 pixels using the robot forward kinematics. An equivalent error was found by reprojecting the grid with new positions of the endoscope corresponding to its displacement across the operating volume. As a consequence, the displacement of the endoscope deduced from the forward robot kinematics will have to be refined by using intra-images measurements to insure the sustain of a precise superimposition. Due to the experimental constraints we could not acquire an angiogram of the dog and test the initialization of the overlay. But the section 2.2 gives a rough estimate of the shift that we can expect. It will have to be interactively compensated by the surgeon after the first superimposition. The right part of the figure 8 illustrates the overlay of a coronary tree in the endoscopic images built off-line from the calibrated endoscopic images and reprojected. Fig. 8. Left: reprojection of the calibration grid using the forward robot kinematics; right: 3D model of the coronary tree built from calibrated endoscopic views. 3 Conclusions and perspectives Despite the difficulties faced in terms of time limitations and logistics, the performed animal experiment was highly rewarding: the operation was suc-

11 Transfer Precision in the Operating Room 11 cessfully carried out according to the transferred planning and the predicted complications were confirmed. It also gave strong support to many of the assumptions built during previous experiments on a plastic phantom, such as the adequacy of the geometric approximations used for the robot and the surrounding obstacles, and unveiled several subtleties that were not accounted for, such as the effect of using a rigid transformation for the registration. Moreover, the time requirements of the transfer methods, summing to under 20 min for any combination, were satisfactory. Efforts in the near future will focus on analyzing the collected data and correlating the observed errors to the transfer steps and assumptions. Furthermore, the effect of the latter on the planned results and the final outcome of the intervention should be clearly identified. New solutions will also have to be sought to increase intraoperative precision, and deal with the motion of internal organs through tracking techniques for use with augmented reality. The long term goal of the analysis work is to adapt our CIS system for planning surgical robotic interventions to the settings of the operating room and its inherent surgical requirements, in order to achieve maximum patient benefit. References 1. L. Adhami and È. Coste-Manière. Positioning tele-operated surgical robots for collision-free optimal operation. In Proc. of the 2002 IEEE International Conference on Robotics and Automation, May L. Adhami and È. Coste-Manière. A versatile system for computer integrated mini-invasive robotic surgery. In Medical Image Computing and Computer Assisted Intervention (MICCAI 02), to appear, L. Adhami, È. Coste-Manière, and J.-D. Boissonnat. Planning and simulation of robotically assisted minimal invasive surgery. In Proc. Medical Image Computing and Computer Assisted Intervention (MICCAI 00), volume 1935 of Lect. Notes in Comp. Sc Springer, Oct F. Devernay, O. Bantiche, and È. Coste-Manière. Structured light on dynamic scenes using standard stereoscopy algorithms. Research report, ChIR Medical Robotics group. INRIA Sophia Antipolis, March F. Devernay, F. Mourgues, and È. Coste-Manière. Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery. In IEEE, editor, Proc. Intl. Workshop on Medical Imaging and Augmented Reality, pages 16 20, O. Faugeras. Three-Dimensional Computer Vision. MIT Press, F. Mourgues and È. Coste-Manière. Flexible calibration of robotized stereoscopic endoscope for overlay in robot assisted surgery. In Medical Image Computing and Computer Assisted Intervention (MICCAI 02), to appear, R. Y. Tsai. A versatile camera calibration technique for high accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation, pages RA 3(4): , August Z. Zhang. Iterative point matching for registration of freeform curves and surfaces. International Journal of Computer Vision, pages 13(2): , 1994.