Towards deformable registration for augmented reality in robotic assisted partial nephrectomy

Transcription

1 Towards deformable registration for augmented reality in robotic assisted partial nephrectomy Supervisor: Elena De Momi, PhD Co-supervisor: Dott. Ing. Sara Moccia Author: Anna Morelli, Academic Year

2 Partial nephrectomy 1 Kidney tumor 12 th most common type of cancer with 338,000 new cases worldwide annually diagnosed

3 Partial nephrectomy 1 Open surgery Kidney tumor 12 th most common type of cancer with 338,000 new cases worldwide annually diagnosed

4 Partial nephrectomy 1 Open surgery Kidney tumor 12 th most common type of cancer with 338,000 new cases worldwide annually diagnosed Minimally Invasive Surgery (MIS) Bleeding reduction Scar and pain reduction Lower risk of infection [Fullum et al., 2010][Himal et al., 2012]

5 Partial nephrectomy 1 Open surgery Kidney tumor 12 th most common type of cancer with 338,000 new cases worldwide annually diagnosed Minimally Invasive Surgery (MIS) Bleeding reduction Scar and pain reduction Lower risk of infection [Fullum et al., 2010][Himal et al., 2012] Robotic MIS (RMIS) 3D view Tremor filtering Instruments wide range of motion

6 Augmented Reality in RMIS 2 RMIS limitations: Lack of visibility High surgeon s cognitive load

7 Augmented Reality in RMIS 2 RMIS limitations: Lack of visibility High surgeon s cognitive load Augmented reality (AR) Benefits: Intra-operative guidance Rapid identification of targets Reduction of surgeon s cognitive load Virtual element (patient s anatomy) Intra-operative view

8 Augmented Reality in RMIS 2 RMIS limitations: Lack of visibility High surgeon s cognitive load Augmented reality (AR) Benefits: Intra-operative guidance Rapid identification of targets Reduction of surgeon s cognitive load The AR should deals the intra-operative soft tissues deformation, mainly due to: Changes in pressure and patient's position Clamping of the renal vessels Surgeon's organ manipulation and dissection [Bernhardt et al., 2017] Virtual element (patient s anatomy) Intra-operative view

9 Registration 3 Registration is the process by which the transformation (i.e. the mathematical operator) necessary to superimpose a given point set to another one is found [Sonka et al., 2002]. Reference point set (F) Metric Registration output Moving point set (M) Transformation Optimization

10 Registration 3 Registration is the process by which the transformation (i.e. the mathematical operator) necessary to superimpose a given point set to another one is found [Sonka et al., 2002]. Reference point set (F) Metric Registration output Moving point set (M) Transformation Optimization Rigid Deformable

11 AR systems in partial nephrectomy 4 Rigid registration 1. Pre-operative anatomy and intra-operative view [Benincasa et al., 2008] [Su et al., 2009] [Glisson et al., 2011] [Pratt et al., 2012]. 2. Pre-operative anatomy modified with biomechianical model and intraoperative view [Altamar et al., 2011] [Ong et al., 2008]. 3. Intra-operative anatomy and intra-operative view [Baumhauer et al., 2008] [Teber et al., 2009] [Cheung et al., 2009] [Nakamura et al., 2010][Hughes-Hallett et al., 2014].

12 AR systems in partial nephrectomy 4 Rigid registration 1. Pre-operative anatomy and intra-operative view [Benincasa et al., 2008] [Su et al., 2009] [Glisson et al., 2011] [Pratt et al., 2012]. 2. Pre-operative anatomy modified with biomechianical model and intraoperative view [Altamar et al., 2011] [Ong et al., 2008]. 3. Intra-operative anatomy and intra-operative view [Baumhauer et al., 2008] [Teber et al., 2009] [Cheung et al., 2009] [Nakamura et al., 2010][Hughes-Hallett et al., 2014]. Deformable registration Deformable superimposition between pre-operative anatomy and intraoperative view is computed only for the tumor boundary [Amir-Khalili et al., 2013].

13 Registration: rigid vs deformable 5 Rigid registration Transformation is described by 6 degrees of freedom in 3D (roto-translation). Most used algorithm is Iterative Closest Point (ICP) [Besl et al., 1992] [Zhang et al., 1994].

14 Registration: rigid vs deformable 5 Rigid registration Transformation is described by 6 degrees of freedom in 3D (roto-translation). Most used algorithm is Iterative Closest Point (ICP) [Besl et al., 1992] [Zhang et al., 1994]. Deformable registration Transformation is described by unlimited number of degrees of freedom [Sotiras et al., 2013]. A possible transformation is Vector Field (VF) [Sederberg et al., 1986] [Rueckert et al., 1999]. VF can be computed with: Thin plate spline Free Form Deformation (FFD) based on B-spline i. No one-to-one correspondence ii. Each point is affected by a limited area iii. Smooth deformation

15 Aim of the thesis 6 The aim of this thesis is exploiting deformable registration in order to obtain an accurate superimposition of the pre-operative kidney anatomy (reconstructed from pre-operative images) to the intra-operative endoscopic view. Superimposition

16 Methods: Workflow 7 PRE-OPERATIVE MODEL GENERATION Segmentation 3D reconstruction Patient s CT Model (M) INTRA-OPERATIVE POINT CLOUD RECONSTRUCTION Initial alignment Deformable registration 3D reconstruction Stereoimages Point-cloud (F)

17 Methods: Pre-operative model generation 8 Segmentation 3D reconstruction Patient s CT Model (M) Pre-operative model generation: The segmentation of the pre-operative patient s CT was performed with semiautomatic method exploiting deformable active contours model [Kass et al., 1988] [Gao et al., 2012]. The 3D reconstruction was performed with marching cubes [Lorensen et al., 1987]. The model vertexes are retrieved. Model generation was implemented in 3D Slicer [

18 Methods: Intra-operative point cloud reconstruction 9 3D reconstruction Stereoimages Point-cloud (F) Intra-operative point cloud reconstruction: The reconstruction was performed with dense soft-tissue 3D reconstruction [Penza et al., 2016]. 3D reconstruction was implemented in C++.

19 Methods: Initial alignment 10 Manual The rotation angles and the translation values were set manually. It was performed with 3D Slicer.

20 Methods: Initial alignment 10 Manual The rotation angles and the translation values were set manually. It was performed with 3D Slicer. Pair-point matching The rigid transformation was retrieved minimizing a cost function based on the Euclidean distance between the corresponding markers. It was implemented in C++ using the Visualization Toolkit (VTK) [ Pre-operative point cloud Intra-operative point cloud Corresponding markers Initial situation Alignment

21 Methods: Deformable registration 11 The deformable registration was computed with Free Form Deformation (FFD) algorithm. 3 3 T x, y = B l u B m v φ i+l,j+m l=0 m=0 B 0 u = (1 u) 3 /6 B 1 u = (3u 3 6u 2 + 4)/6 B 2 u = ( 3u 3 + 3u 2 + 3u + 1)/6 B 3 (u) = u 3 /6 and φ i+l,j+m, are the mesh control points: i = x/n x 1 j = y/n y 1 u = x/n x x/n x v = y/n y y/n y The FFD registration algorithm was implemented in C++ using the Insight Segmentation and Registration Toolkit (ITK) [

22 Methods: Phantom development 12 In order to retrieve intra-operative point cloud. Moulding process [Condino et al., 2011]: 1. From the pre-operative model, the 3D virtual negative molds were modeled with Blender. 2. The mold were printed in acrylonite butadine styrene (ABS) 3. Phantoms were built in bi-component silicon elastomer (Pro-lastix); silicon oil was added to make the phantoms deformable and color was chosen to mimic the real texture

23 Materials: Pre-operative point cloud 13 M1: from Ircad dataset [ Abdominal CT, 512x512x167 slices and 0.916x0.196x1.8 mm voxel size

24 Materials: Pre-operative point cloud 13 M1: from Ircad dataset [ Abdominal CT, 512x512x167 slices and 0.916x0.196x1.8 mm voxel size. M2: from CT given by Istituto Europeo di Oncologia Abdominal CT, 512x512x716 slices and 0.703x0.703x0.625 mm voxel size.

25 Materials: Intra-operative point cloud 14 F1: manually deforming M1 with Blender [

26 Materials: Intra-operative point cloud 14 F1: manually deforming M1 with Blender [ F2: from images acquired on kidney phantom

27 Evaluation protocol 15 To test the registration performance, the Root Mean Square Error (RMSE) reduction before the registration (RMSE initial ) and after (RMSE final ) were compared: Δ RMSE = RMSE initial RMSE final RMSE initial where the RMSE= d(m, F) and d(m, F) = σ m M min Ԧf P Ԧf m.

28 Evaluation protocol 15 To test the registration performance, the Root Mean Square Error (RMSE) reduction before the registration (RMSE initial ) and after (RMSE final ) were compared: Δ RMSE = RMSE initial RMSE final RMSE initial where the RMSE= d(m, F) and d(m, F) = σ m M min Ԧf m. Ԧf P The Wilcoxon test (significance level = 0.05) was used to investigate the presence of statistically significative difference.

29 Evaluation protocol 15 To test the registration performance, the Root Mean Square Error (RMSE) reduction before the registration (RMSE initial ) and after (RMSE final ) were compared: Δ RMSE = RMSE initial RMSE final RMSE initial where the RMSE= d(m, F) and d(m, F) = σ m M min Ԧf m. Ԧf P The Wilcoxon test (significance level = 0.05) was used to investigate the presence of statistically significative difference. A multicomparison on the means was performed to investigate the presence of differences in the registration results [Hochberg, 1984].

30 Evaluation protocol 15 To test the registration performance, the Root Mean Square Error (RMSE) reduction before the registration (RMSE initial ) and after (RMSE final ) were compared: Δ RMSE = RMSE initial RMSE final RMSE initial where the RMSE= d(m, F) and d(m, F) = σ m M min Ԧf m. Ԧf P The Wilcoxon test (significance level = 0.05) was used to investigate the presence of statistically significative difference. A multicomparison on the means was performed to investigate the presence of differences in the registration results [Hochberg, 1984]. The used PC had AMD Ryzen 7 processor and 32 GB of RAM.

31 Evaluation protocol 16 Objective n x, n y and iter max FFD vs ICP and deformation levels Entire workflow Description In 2D, the model M1 was registered to the manually deformed F1. In 2D and 3D, the deformable registration was performed between M1 and the manually deformed F1. The models M1 and M2 were registered to F2 (the intra-operative point cloud acquired on phantom).

32 Results: n x, n y and iter max 17 n x, n y Δ RMSE dim x /5, dim y / dim x /15, dim y / dim x /20, dim y /

33 Results: n x, n y and iter max 17 n x, n y Δ RMSE dim x /5, dim y / dim x /15, dim y / dim x /20, dim y /

34 Results: FFD, ICP and deformation level in 2D 18 Intra-operative point cloud Pre-operative point cloud Initial situation Δ RMSE,ICP = 0.02 Δ RMSE,FFD = 0.51

35 Results: FFD, ICP and deformation level in 2D 18

36 Results: FFD, ICP and deformation level in 3D 19 Intra-operative point cloud Pre-operative point cloud Initial situation Δ RMSE,ICP = 0.01 Δ RMSE,FFD = 0.12

37 Results: FFD, ICP and deformation level in 3D 19

38 Results: Entire workflow 20 Initial situation Alignment Registration Initial manual alignment: Mean Δ RMSE,workflow : 0.74 Best Δ RMSE,workflow : 0.91 Pre-operative point cloud Intra-operative point cloud Point-based alignment: Mean Δ RMSE,workflow : 0.78 Best Δ RMSE,workflow : 0.87 Corresponding marker 000

39 Conclusions 21 The workflow is able to treat in quasi-real time the deformable superimposition of the 3D kidney model to the intra-operative point cloud Maintain the superimposition: tracking [Bernhardt et al., 2017] [Yip et al., 2012] Deal with the amount of visible surface

40 Conclusions 21 The workflow is able to treat in quasi-real time the deformable superimposition of the 3D kidney model to the intra-operative point cloud Maintain the superimposition: tracking [Bernhardt et al., 2017] [Yip et al., 2012] Deal with the amount of visible surface The manual intervention in necessary in three phases: 1. Pre-operative CT segmentation Automatic CT segmentation [Lee et al., 2003] [Lin et al., 2006] [Yan et al., 2010] 2. Manual initial alignment 3. Markers selection

41 Conclusions 21 The workflow is able to treat in quasi-real time the deformable superimposition of the 3D kidney model to the intra-operative point cloud Maintain the superimposition: tracking [Bernhardt et al., 2017] [Yip et al., 2012] Deal with the amount of visible surface The manual intervention in necessary in three phases: 1. Pre-operative CT segmentation Automatic CT segmentation [Lee et al., 2003] [Lin et al., 2006] [Yan et al., 2010] 2. Manual initial alignment 3. Markers selection Implementation of Active-Constraints

42 Thank you for your attention!

43 Appendix The deformable registration was computed with Free Form Deformation (FFD) algorithm T x, y, z = B l u B m v B n w φ i+l,j+m,k+n l=0 m=0 n=0 B 0 u = (1 u) 3 /6 B 1 u = (3u 3 6u 2 + 4)/6 B 2 u = ( 3u 3 + 3u 2 + 3u + 1)/6 B 3 (u) = u 3 /6 and φ i+l,j+m,k+n are the mesh control points: i = x/n x 1 j = y/n y 1 k =[z/ n z ] 1 u = x/n x x/n x v = y/n y y/n y w = z/n z z/n z