Physiological Motion Compensation in Minimally Invasive Robotic Surgery Part I

Physiological Motion Compensation in Minimally Invasive Robotic Surgery Part I Tobias Ortmaier Laboratoire de Robotique de Paris 18, route du Panorama - BP 61 92265 Fontenay-aux-Roses Cedex France Tobias.Ortmaier@alumni.tum.de

Overview Introduction - Minimally invasive surgery - Minimally invasive robotic surgery Motion compensation in beating heart surgery - Manual manipulation limitations - Motion measurement with artificial markers - Motion measurement with natural landmarks Predicting the heart motion - Local prediction - Global prediction - Sensor fusion Conclusion and Outlook 2

What is Minimally Invasive Surgery? Surgeon works with long instruments through small incisions and uses laparoscope for visual feedback. 3

What is Minimally Invasive Surgery? Schematic overview - pecularities Reverse hand motion Scaling of velocity Loss of degrees of freedom 4

What is Minimally Invasive Surgery? Properties Advantages - Less trauma - Shorter hospital stay - Cosmetical advantages Disadvantages - Reduced sight - Trocar kinematics - Reduced haptic feedback - Long training period 5

Minimally Invasive Robotic Surgery Telepresence System Instruments are not manipulated directly by the surgeon but by specialized robotic arms 6

Existing MIRS Systems No haptic feedback No full manipulability inside patient Scaling of motion Optical magnification Modular system Computer Motion Inc. - Zeus 7

Existing MIRS Systems No haptic feedback Full manipulability inside patient Scaling of motion Optical magnification Monolithic system Intuitive Surgery Inc. - davinci 8

Existing MIRS Systems 7 Degrees of Freedom Small Size Robot University of Tokyo Small 7 degrees of freedom surgical robot Sterilizable instruments Passive system to position the surgical robot Prof. Nakamura, Univ. Tokyo 9

Fields of Application Radiosurgery - Tumor motion is measured with stereo X-ray cameras - Companies: BrainLab (Novalis) and Cyberknife Liver biopsy - Needles are guided towards the moving lesions - Carnegie Mellon University (Riviere et al) and University of Tokyo (Hong et al) Heart valve repair - Moving valve is automatically grasped - GABIE project in the French ROBEA program Bypass grafts - Motion of beating heart is compensated - German Aerospace Center, LSIIT Strasbourg, Univ. of Tokyo 11

Fitt s Law Describes the difficulty to touch a (non-moving) goal Index ID [bit] for movement difficulty: ID = log 2 (2 A / W) with A: amplitude of intended movement W: target width Time needed to complete task: T m = k m + c m ID with k m : constant, being approx. 0.178 s c m : measure of information handling capacity, being approx. 0.1 s / bit With increasing distance the necessary time increases 12

Modified Fitt s Law Modification of Fitt s law for moving targets, proposed by Jagacinski et al Time needed to complete task: with T m = c + d A + e (V + 1) [(1 / W ) - 1] c, d, e: constant coefficients A: amplitude of intended movement W: target width V: target velocity With increasing velocity the necessary time increases 13

Manual Control and Tracking Studies of manual tracking performance indicate a bandwidth estimation of about 1 Hz. Therefore, a heart rate of 1 Hz (i.e. 60 beats / min) lies at natural limit. Manual position accuracy is limited to 0.1 mm to 0.2 mm. Wait and see or locking strategy not applicable due to irregular motion. Manual tracking experiments were carried out at Heartcenter Leipzig to further investigate influence of motion. Falk, MD, PhD., Leipzig 14

Manual Control and Tracking Experimental set-up Falk, MD, PhD., Leipzig 15

Manual Control and Tracking Time to touch two targets in a horizontal line Distances A1 < A2 < A3 Frequencies 35 and 50 / min Amplitudes: st = stabilized nst = nonstabilized Experimental Results Error rates for complex pattern touch task With higher frequency and larger amplitude error rate increases Falk, MD, PhD., Leipzig 16

Minimally Invasive Beating Heart Surgery Gentle form of surgery No heart-lung machine Heart Motion Mechanical stabilizer Remaining motion (motion 1.5 mm 2.0 mm; frequencies up to 2 Hz) Long operation duration Additional training does not improve quality of surgery Solution: Motion compensation in minimally invasive beating heart surgery 17

Minimally Invasive Beating Heart Surgery 18

Overview of Research in Motion Compensation in Beating Heart Surgery Motion compensation with - artificial landmarks (i.e. markers), - 500 Hz highspeed camera, - and adaptive filtering. - Performed at LSIIT Strasbourg, by M. de Mathelin - Detailed in next presentation Virtual stillness - Performed at University of Tokyo, by Y. Nakamura - Detailed in this presentation Motion compensation with natural landmarks - German Aerospace Centre - Detailed in this presentation 19

Virtual Stillness Nakamura, Univ. Tokyo Experimental setup - Overview Artificial markers to track heart motion High speed camera (955 frames per second) Visual stabilization Small 7 degrees of freedom surgical robot Sterilizable instruments Passive system to position the surgical robot Motion compensation 20

Visual Stabilization Stabilization of point of interest by cutting out, moving, and zooming area of interest Prof. Nakamura, Univ. Tokyo 21

Motion Compensation In vivo experiment on a beating pig heart Prof. Nakamura, Univ. Tokyo 22

Motion Estimation with Natural Landmarks Properties - Limited space - Strongly nonlinear motion of tissue - No motion model available Natural landmarks - Prominent image features - Intensity based approach - Local motion model - Image preprocessing is necessary (illumination and specularities) 23

Motion Tracking Scheme Heart Motion 24

Specular Reflections - Detection Arise on heart surface from directed light source Irregular occurrence disrupts tracking Distinctive intensity allows Detection by thresholding Fast and robust approach 25

Specular Reflections - Detection Thresholding I = I offset TH + Dilation twice S = 0 1 0 1 1 1 0 1 0 I(image) 26

27 Specular Reflections - Structure Detection Structure Tensor ( ) = = 2 2 ) ( ) ( * * x f y f x f y f x f x f g f f g J σ σ σ σ σ σ ρ σ σ ρ ρ with Gradient of Gaussian smoothed image Tensor product Integration scale Noise scale f σ ρ σ

Specular Reflections - Reconstruction Algorithm Calculate eigenvalues of structure matrix : J λ1 λ2 Eigenvector corresponding to smaller eigenvalue ( λ 2 ) gives structure orientation Determine boundary of specularity in structure direction Linear interpolation boundary points Final Gaussian smoothing ρ 28

Specular Reflections - Reconstruction Algorithm Principle Example 29

Tracking Results Original sequence with specularities Reduced affine motion model with sum of squared (SSD) differences as similarity measure applied 30

Tracking Results Structure tensor based reconstruction Reduced affine motion model with sum of squared (SSD) differences as similarity measure applied 31

Tracking Results Landmark trajectories 32

Tracking Results Amplitude spectrum Dominant peaks Respiration frequency (f 1 =14.5 1/min) Heart rate (f 2 =70.9 1/min) 33

Motivation What happens in case of occlusions? Tracking fails! Solution: Motion prediction 35

Motion Prediction Scheme Prediction of heart motion based on Quasi-periodicity of trajectories Additional signals (ECG and RPS) Model for heart motion difficult Patient specific Dependent on mechanical stabilizer position Unknown model parameters Solution: Model free prediction scheme 36

Local Prediction Exploitation of quasi-periodicity of trajectories Prerequisites Smooth trajectories No transient inner states 37

Motion Prediction Scheme Takens Theorem: Taking a sufficiently long vector built of past values of a time series enables the reconstruction of the underlying structure of the system dynamics which produced the sequence. Consequence: Two ways to describe a dynamic system: explicit model appropriate embedding vectors 38

Local Prediction + Dynamic positioning of tracking search area + Compensation of outliers - Only short disturbances 39

Five-Step Prediction + More step prediction is possible - Prediction accuracy deteriorates 40

One-step prediction Local Prediction - Summary Application of Takens Theorem Detection and compensation of outliers Positioning of search window More-step prediction With increasing prediction interval prediction accuracy deteriorates 41

Global Prediction Not all LMs are simultaneously occluded Use visible LMs for prediction of occluded LMs Prerequisite: functional relationship between trajectories Result: Global prediction scheme 42

Global Prediction Simultaneous exploitation of several landmarks Prerequisite: functional relationship between trajectories 43

Global Prediction Position of landmarks 44

Global Prediction + Longer disturbances + More robust than local scheme - Estimation depends on visual information only 45

Global Prediction - Summary Application of Takens Theorem Robust long-term prediction Combination of local and global prediction scheme possible Compensation for long occlusions What happens if vision fails? 46

ECG and RPS Based Prediction Correlation of signals ECG, RPS, t x, t y Correlation 47

Algorithm and Landmarks Similar to global prediction ECG and RPS as additional LM 48

Global Prediction - Results Predicted trajectory Prediction error 49

Conclusion Minimally invasive surgery offers significant benefits for the patient Physiological organ motion deteriorates surgeon s performance In MIRS systems organ motion can be compensated Then, surgeon works on a virtually stabilized organ Robust organ motion measurement in a clinical environment is crucial Overall system reliability is increased by - Motion prediction - Additional sensor signals 51

Outlook System Overview I 52

Outlook System Overview II 53