Computer-assisted fracture reduction in an orthopaedic pre-operative planning workflow

Transcription

1 LiU-ITN-TEK-A--17/003--SE Computer-assisted fracture reduction in an orthopaedic pre-operative planning workflow Ludvig Mangs Department of Science and Technology Linköping University SE Norrköping, Sweden Institutionen för teknik och naturvetenskap Linköpings universitet Norrköping

2 LiU-ITN-TEK-A--17/003--SE Computer-assisted fracture reduction in an orthopaedic pre-operative planning workflow Examensarbete utfört i Medieteknik vid Tekniska högskolan vid Linköpings universitet Ludvig Mangs Handledare Peter Steneteg Examinator Martin Falk Norrköping

3 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: Ludvig Mangs

4 Computer-assisted Fracture Reduction in an Orthopaedic Pre-operative Planning Workflow Ludvig Mangs Master of Science Thesis in Media Technology Department of Science and Technology, Linköping University, 2017

5 Master of Science Thesis in Media Technology Computer-assisted Fracture Reduction in an Orthopaedic Pre-operative Planning Workflow Ludvig Mangs LiU-ITN-TEK-A--17/003--SE Supervisor: Examiner: Principal Research Engineer, Peter Steneteg itn, Linköping University Mattias Bergbom Sectra Assistant Lecturer, Martin Falk itn, Linköping University Scientific Visualization and Human Computer Interaction Department of Science and Technology Linköping University SE Norrköping, Sweden Copyright 2017 Ludvig Mangs

6 Abstract There are many different types of bones with varying characteristics in the human body. In case of fractured bones it is necessary for an orthopaedic surgeon to be abletohandlealargevarietyofproblems. Toavoidaninfectionitispreferredto have a short operation duration and therefore it is important to plan and prepare before opening up the patient. A pre-operative planning is performed by the surgeon, traditionally using a stack of 2D images from body scans, where it is decided how to reduce the fracture. For more complicated fractures it can be difficult to see how to solve thebonepuzzleandinthosecasesitmaybebeneficialtousea3drepresentation of the problem. In this environment it is possible to do a thorough planning by reducing a fracture completely in a virtual environment and recording the process to be able to do the same during the operation. Orthopaedic surgeons most commonly use a mouse and keyboard as navigation when they perform the pre-operative planning. To perform fine positioning and rotation of several bone pieces in 3D can be a tedious task, even for an experienced user. Several papers have been written where algorithms to solve several different cases of fractures using manual, semi-automatic, or completely automatic methods have been tested. This report presents three implementations for solving the bone puzzles described above, with varying levels of required interaction: two semi-automatic ones and one which is automatic. These are compared using qualitative as well as quantitative tests to find out if less interaction can yield equal or better results. Qualitative tests are performed on real clinical data from CT-scans, which have been exported using marching cubes and processed using standard filters in Meshlab. A model created in Blender is used for quantitative tests. Test results have shown that each implementation has its own strengths and weaknesses which can make them usable for different types of fractures. Manual solutions were difficult to use when it comes to a large amount of pieces and the automatic solution did not always yield a satisfactory result. However, it was faster and easier to use the automatic solution compared to the manual ones. It may be possible to combine automatic solutions and manual ones to increase the number of solvable cases. The conclusion is that it is possible to reduce fractures with less user interaction and still get equal or better results, but it depends on the fracture case as well as the user. iii

7

8 Acknowledgments Firstly I want to say thanks to Mattias Bergbom and Stefan Lindholm for providing feedback and for giving me freedom in the shaping of the thesis project. Another thanks goes to my supervisor Peter Steneteg and my examiner Martin Falk for their feedback regarding the project ideas and report content. I want to thank my family for being supportive and helpful during times of doubt and Ikshu Dutta for being a great office roommate and making every day a fun experience. Last but not least I want to thank all the kind people at Sectra for making me feel welcome as well as for all the help and encouraging words I received during my time there. Norrköping, December 2016 Ludvig Mangs v

9

10 Contents 1 Introduction Problem Description Purpose and Approach Delimitations Report Structure Background Prerequisites Orthopaedics Bones and Fractures Computed Tomography Computer Assisted Tools in Orthopaedics Related Work Manual Approaches Automatic Approaches Semi-Automatic Approaches Solving 3D Puzzles Manual Point Pair Alignment Selection of Points Matching the Point Pairs Manual Point Cloud Alignment Adding Point Clouds Matching the Point Clouds Automatic Point Cloud Alignment Procedure Overview Finding Fractured Areas Classifying Fracture Area Points Hierarchical Matching Order Improve Initial Conditions Increase Neighbor Points Generating Bone Meshes vii

11 viii Contents 4 Results A Quantitative Implementation Comparison A Qualitative Implementation Comparison Pelvic Case Tibia Case Algorithm and Parameter Tests Iterative Closest Point Detect and Classify Fracture Points Improve Initial Position Increase Neighbor Points Discussion Implementation Discussion Manual Point Pair Manual Point Cloud Automatic Point Cloud Manual VS Automatic Evaluation Drawbacks Quantitative Evaluation Qualitative Evaluation Future Work Planning Analysis Calculation Optimization Fracture Detection using Face Normals Improved User Interaction ICP Configuration Tests Point Match Weighing Final Adjustments Visual Feedback Transparency Match Bones Using Bone Models Multiple Mesh Resolution Conclusion 49 Bibliography 51

12 1 Introduction Orthopaedics is a medical specialty which focuses on the body s musculoskeletal system. When a person is injured bones may fracture and require an orthopaedic surgeon to perform a fracture reduction surgery which restores bones to their correct alignments. By planning a surgery beforehand errors can more easily be avoided and it may allow the surgeon to finish the surgery faster which decreases the risk for infection. Astheworldhasbecomemoredigitalpeopletrytocomeupwithnewwaysof using available technology within different fields, including orthopaedic surgery. Itmayforexamplebeusedforplanningpriorto,assistingduring,orforanalysis after a surgery. By using a virtual environment for surgery planning it is possible for the orthopaedist to virtually reduce the fracture using computer-assisted tools and get a clear view of what the result will look like. Orthopaedists commonly use computers with traditional interaction methods such as mouse and keyboard. As no transferable surgery skills are gained from performing virtual surgery planning using this kind of interaction, and it therefore cannot be considered as training, the result is what is important. Depending on the fracture type it may be difficult to get a good understanding of how to reduce the fractures using 2D images and because of this Sectra has a virtual 3D planning tool where the bones can be visualized and manipulated. Using interaction tools with a low degree of freedom in three dimensions can make it difficult to handle fine manual adjustments intuitively, which is necessary for a virtual fracture reduction. As the important part is the planning result and manual manipulation can be a tedious task it would be beneficial to minimize the amount of time and effort required by the orthopaedic surgeon during surgery planning. This thesis studies whether it is possible to reduce fractures by using methods requiring less input from the user without it negatively affecting the result when compared to more manual tools. 1

13 2 1 Introduction The contribution of this work to the Orthopaedic pre-operative planning and 3D puzzling is the following: A way of viewing and interacting with individual pieces of the puzzle without affecting their original transform. Classification of automatically found fractured areas using a closest distance method. An automatic iterative matching approach with several pieces where a version of the Iterative Closest Point (ICP) [27] algorithm is combined with updated point clouds. Improvement of initial positioning between pieces using an iterative approach with a middle-point displacement and updated point clouds. Using these methods along with previously documented approaches it was possible to create three implementations which each could be used to solve three different 3D puzzles. Each solver had different strengths and weaknesses and the results varied from case to case. 1.1 Problem Description Sectra s orthopaedic department currently has a pre-operative planning tool where orthopaedists can get a view of bone fractures in 2D and 3D which gives a better understanding of what the problem at hand looks like. It is possible to interact with the bones in the 3D scene using manual tools for translation and rotation as well as a semi-automatic point matching method which uses weighted point pairs for alignment. One problem is that it can be difficult to navigate and interact with a 3D environment using a mouse and keyboard interaction, which is what orthopaedists commonly use. If the planning process could become more automatic it could save much time and effort. 1.2 Purpose and Approach The purpose of this thesis is to investigate if it is possible to simplify the interaction required by the orthopaedists to reduce fractures during 3D surgery planning, while maintaining or improving the overall quality of the final result. This is done by comparing how well three different implementations solve 3D puzzles for three different cases, as well as by discussing other pros and cons with each of the methods. 1.3 Delimitations A complete orthopaedic pre-operative planning consists of several steps as is describedinchapter2anditwouldbeoutsidethescopeofthethesistoincludeall

14 1.4 Report Structure 3 of them. Some delimitations have been decided on to save time and improve the quality and usefulness of the end-result: 1. The 3D puzzle solving will be performed on mesh data. 2. Meshes will be pre-processed using standard algorithms for reducing the vertex-count. 3. As the solution should be usable on standard hospital systems the interaction method is strictly mouse and keyboard. 4. The bone pieces should have a reasonably good initial alignment. 1.4 Report Structure Initially, Chapter 2 describes different aspects of computer-assisted approaches within orthopaedics, as well as what has previously been done within 3D mesh puzzling. Chapter 3 explains the methods used for each implementation as well as which tests will be performed. Results from the tests are then presented within Chapter 4 and later discussed in Chapter 5. In Chapter 6 the thesis conclusions will be presented.

15

16 2 Background This chapter contains information about orthopaedics and computer-assisted tools availablewithinthearea,therelatedworkwithin3dpuzzling,aswellasourcontribution. 2.1 Prerequisites It is expected that the reader has basic knowledge in computer graphics, linear algebra, and programming. 2.2 Orthopaedics Orthopaedics is a medical specialization with focus on diseases and traumas in the musculoskeletal system. There are several different aspects to being an orthopaedic surgeon when it comes to handling different injuries and disorders: diagnosis, treatment, rehabilitation, and prevention, which each can be performed using a variety of approaches [2]. When diagnosing a patient with fractured bones it is important for the orthopaedic surgeon to be knowledgeable in the bone and fracture types involved. Depending on the severity of the problem it may be necessary to get a view of the fractured bones inside the body which is commonly done using computed tomography (CT) Bones and Fractures In the body of an adult human there are 206 bones, a newborn child has 270 but many of them fuse while growing up. There are three different kinds of main 5

17 6 2 Background Figure 2.1: Different types of fractures on a long bone which may affect the surgeon s choice of treatment method [13]. bones: long bones, flat bone and short bones which each have different characteristics[7]. An orthopaedic surgeon is knowledgeable of the whole musculoskeletal system but they can also specialize in one or several areas, for example ankles or spines, or in specific fields such as trauma [2]. Threeofthemostcommonreasonsforbonestofracturearetraumawhichcan for example be due to some kind of accident, osteoporosis which weakens bones, and overuse because of repetitive motions. There are also several ways for bones to break depending how far across the breaking point the bone has been forced. Itcouldbeasmallcrackoritcouldcausethebonetoshatterindifferentways[1], as shown in Figure 2.1. This means that there are many different combinations of bone and fracture types which could affect the choice of treatment method Computed Tomography When orthopaedic surgeons evaluate a fracture on a patient the most common method is to use X-rays to get a clear view of the bones [1]. For this a method called computed tomography (CT) is commonly used. It works by sending a large amount of X-rays through the body from different angles and measuring their attenuations. Using these measurements a computer can construct images of cross-sections of the body interior. These can be used to create a three dimensional representation of the body [11]. To be able to make sense out of the numbers from the CT scan Hounsfield units(hu) can be used. It is a linear transformation of the attenuation coefficients with respect to the radiodensity of air and water [4]. Using the Hounsfield scale it is possible to label body content such as fat, tissue, and bone, as can be seen in Figure 2.2. To get numbers which are more relevant to true absorption, 1000 shouldbeaddedtothesevalues,whichwouldchangethevalueofairtozeroand water to 1000 [11]. 2.3 Computer Assisted Tools in Orthopaedics There are several different ways of using computers to assist orthopaedists as described in the survey by Jimenez-Delgado et al. [13]. There are some which can

18 2.3 Computer Assisted Tools in Orthopaedics 7 Figure 2.2: The Hounsfield scale with set values for air, water, and a few body composition components [11].

19 8 2 Background Figure 2.3: An overview of computer-assisting tools in orthopaedic surgery wherethedottedgraysectionsareoutsidethescope ofthis thesiswork[13]. be useful before surgery such as planning and training, tools which can be used during a surgery like aiding and tele-operating, and finally computers can be used to analyze the results from an operation afterwards, see Figure 2.3. Within this thesis the focus will be on the part of the image which is not faded out: presurgery planning. Pre-operative planning can be divided into three sub-categories as well, as shown in Figure 2.4. To perform a complete pre-surgery planning using CT images one must perform these three steps described in short below [13]: Generation of bone fragment models which means using patient data to generate data models which are useful for the task at hand. Typically volume data for visualization, point clouds for efficient interaction, and meshes for geometric operations. Virtual bone fracture reduction planning where the goal is to use different methods to align the bone fragments with each other in a correct manner to reduce the fracture. Also included is to plan the stabilization steps. This can be done using fixation devices such as the one shown in Figure 2.5. Analysis of a virtual bone fracture reduction planning where the result or the procedure performed in the fracture reduction planning is analyzed for evaluation purposes. The original bone fragment models can also be analyzed in this step. Insomecasestherearesolutionswhichcandoallofthesestepsonallkindof bone fractures and others are more focused on a single problem. 2.4 Related Work The thesis is about developing tools for solving a 3D puzzle to reconstruct an object. Thisisdonebyfindingthetranslationsandrotationsfortheobjectsforan optimal alignment. There have been several papers written on this subject using

20 2.4 Related Work 9 Figure 2.4: An overview of the different stages of a virtual pre-operative planning using CT images. The numbers in the figure correspond to the original article s section for each part [13]. Figure 2.5: A fixation device on a broken bone model. The fixation is depicted in gray, while the different bone fragments are highlighted in different colors [12].

21 10 2 Background different approaches, within orthopaedics as well as within other areas such as archaeology. In most cases several levels of automacy is used in different steps to achieve the end result Manual Approaches In some of the related work completely manual approaches use other navigation devices than mouse and keyboard for rotation and translation. Fornaro et al. [8] propose an approach where a haptic device with haptic feedback is used on a pelvis fracture. Mellado et al. use an electromagnetically tracked device on broken archaeology objects [19]. Lee et al. [17] use an approach for manual reconstruction using mouse and keyboard interaction [16]. Each bone piece is transformed by three rotations and two translations along the viewing plane. They mention that the operation time can be long when performing this procedure on many fragments Automatic Approaches When it comes to completely automatic approaches the first step is usually to improve the initial position if needed and after that using algorithms to either align the pieces fractured areas with each other or aligning the pieces with a predefined model of the reconstructed object. Reference Model Alignment Whenitisknownwhattheobjectlookslikewhenitisnotbroken,itispossibleto usethatknowledgetohelpwiththealignmentofthebrokenfragments. Itcanbe used both for rough initial alignment but also to completely reduce fractures in some cases. Fürnstahl et al. [9] propose the use of a mirrored contralateral bone to improve initial alignment of bone pieces. Okada et al. [20] use this method for a complete fracture reduction, but conclude that the results improve when combined with another alignment method. Thomas et al. [23] use a predefined intact bone template for both the rough and the fine alignment of fractured pieces. Albrecht et al. [5] use a statistical bone model as they lacked the contralateral bone and require a good initial position for thebonealignment. Adrawbackisthatthelengthoftheboneneedstobedefined by the user for a good result. Fracture Area Alignment When there is a puzzle but no known solution the idea is to find the fractured areas and align them with each other to solve the puzzle. This is often done with the following steps: 1. Finding fractured areas on meshes. 2. Adding point clouds to these areas.

22 2.4 Related Work Aligning the pieces using a point match algorithm. Several papers propose solutions for automatically finding areas or lines of interest. Okada et al. [20] propose a technique for finding fracture lines (FL) by performing a curvature analysis on volume data. Willis et al. [26] make use of Hounsfield values from the CT scan to separate cortical and cancellous bone to find the fractured areas of the bone pieces. In this report the bone pieces will be used in the form of meshes which means that above mentioned methods are not directly applicable. Fürnstahl et al. [9] however use a method to find FLs on mesh data by comparing each vertex s normal direction with the ones of nearby vertices and is thus more relevant. After the meshes interesting parts have been found they need to be aligned in an optimal way. To represent the surfaces which should be aligned the most common method is by using point clouds. The points can be applied randomly along the surface or by considering the object s geometry as proposed by Gelfand et al. [10]. The final step is to use the point clouds to align the bone pieces and reduce the fracture using a point matching algorithm, which can be done in several ways. In the survey by Jiménez-Delgado et al. [13] all of the papers using this type of matching make use of variations of an algorithm called Iterative Closest Point (ICP). It iteratively matches each point in a point cloud with the closest point in another cloud. This is a commonly used method within 3D puzzling but is also known to get stuck in local minima if initial conditions are bad. Fürnstahl et al. [9] claim that the use of a kernel correlation-based registration described by Tsin et al. [25], which matches a point to several points in the other mesh, will lower the risk of finding local minima. Multi-Piece Alignment When aligning the pieces with each other there may be pair-wise and group-wise issues to consider depending on the number of pieces to be aligned as mentioned in Zhou et al. [28]. In the paper it is suggested to iteratively add a new mesh to an active set of mesh pieces, starting with the ones with most matching edges. When a new member is added to the group it is paired with the already added meshes which are fixed in place. After each iteration the group is re-aligned to adapt to the new fragment. First pair-wise points are calculated to be matched on each of the pieces and is followed up by a multi-piece alignment. When aligning several pieces the errors between each pair may accumulate and thus an ideal mates concept which is described by Pulli is used [21]. A final global alignment is then used by iteratively adding bone fragments to a set of active meshes and matching them with the already existing pieces Semi-Automatic Approaches When it comes to more complicated puzzles with many pieces, and maybe some pieces missing, it may become difficult to apply an automatic solution to solve the

23 12 2 Background problem. That, as well as it being a tedious task to perform a completely manual reconstruction of the fractured object has led to there being several papers which suggests semi-automatic approaches. It is often claimed that by combining automatic calculations with the knowledge of the orthopaedist user the solutions may become more generally applicable. Manual Rough Alignment Asthepiecesmaynotalwaysbeinagoodinitialpositionandrotation,especially not when it comes to archaeological problems there may be a need to help the automatic matching algorithm by improving the initial position. Mellado et al. [19] combine manual alignment with an ICP algorithm which in real-time finds the optimal fit with the nearby pieces. Fracture Area Alignment When it comes to performing an automatic fracture area alignment the issue can be to come up with an algorithm for finding areas of interest for a general bone. Semi-automatic approaches commonly use similar matching algorithms as the automatic approaches but instead use manual tools to mark the bone fragments. Willisetal. [26]findsthefracturedareasautomatically,butleavesituptothe user to select regions on the different pieces which are likely to correspond with eachother. Andreadisetal. [6]haveauserinterfacewheretheusercanmarkthe salient features of pieces which share similar features and might be good matches. Zhou et al. [28] use surface patches defined by the user which are automatically filled with points, either randomly or with respect to geometry. Lee et al. [17] suggests using a multi-point positioning where point pairs are added to bones and matched using a coordinate transformation algorithm.

24 3 Solving 3D Puzzles In this chapter three implementations which can be used to solve 3D puzzles are presented along with descriptions of their respective algorithms. Each of them has a different level of automacy and will be tested to see whether a manual or an automatic approach is superior when it comes to 3D puzzling. 3.1 Manual Point Pair Alignment Thefirstoftheimplementationsistheonewhichreliesmostoncorrectuserinput. A number of point pairs on two meshes are selected and the pieces are aligned automatically. For the step-by-step overview, see Figure Selection of Points When selecting points the user may choose one moving object and then place pairsofpointsonthemovingobjectandtheotherpiecesinthescene. Forclarity the points will be colored according to which pair they belong to, see Figure 3.2. Isolation Mode As the bone pieces are close together after a fracture it may be difficult to study and apply manual tools to individual bone fragments. To be able to get a closeup of pieces where it is easier to mark fractured areas an Isolation Mode has been added. This will move the piece closer to the camera and allow the user to rotate and translate it to get a clear overview of each of the pieces. When the pieces are removed from the isolation they return to the state they had before being isolated. It is also used for the Manual Point Cloud implementation in Section

25 14 3 Solving 3D Puzzles (a) Initial positions. (b) Isolate the pieces. (c) Rotate to view fractured areas. (d) Add point pairs to the pieces. (e) Disable the isolation mode. (f) Perform alignment. Figure 3.1: Manual Point Pair alignment performed on two pieces of a clinical pelvic case Matching the Point Pairs As the user has already defined which pairs of points to use for the matching what is needed to perform the alignment is an algorithm which aligns a number of point pairs. This can be done using a variant of an algorithm called Kabsch algorithm [14][15]. It finds the optimal rotation matrix which minimizes the root mean squared deviation (RMSD) between two paired point sets. The sets of points are in this case represented as a Nx3 matrix, but the algorithm is also usable for general cases with NxM dimensions. It is executed using the following steps: Translation First it is necessary to translate the centroids of each set of points to the origin. Thisisdonebysubtractingthecentroidpositionfromeachpointintheset. Finding the centroid of a point set is done by calculating the average position of all the points in the shape as shown in equation 3.1.

26 3.1 Manual Point Pair Alignment 15 Figure 3.2: A visualization of how points are paired using different colors. The colored lines between the pairs are used for clarification and are not included in the actual implementation. C = 1 N N p k (3.1) Where C is the centroid for the point set, N is the number of points in the set and p k is a point in the set. Covariance Matrix Secondly a covariance matrix is calculated using equation 3.2 k=1 A = P T Q (3.2) where A is the covariance matrix, P T is the first set of points and Q is the second set of points in matrix form. They can also be represented using a summation form as shown in equation 3.3. A ij = N P ki Q kj (3.3) k=1 Computation of the Optimal Rotation Matrix The optimal rotation matrix U can be calculated using U = (A T A) 1/2 A 1, but this is a complicated calculation when considering special cases like a lack of the

27 16 3 Solving 3D Puzzles inverse matrix A 1. A different solution is to use a singular value decomposition (SVD) which makes it possible to simplify the algorithm. First the SVD is calculated from the covariance matrix A as shown in equation 3.4. For this a.net linear algebra library called Mapack is used [3]. A = VSW T (3.4) Where V and W areunitarymatrices, W T istheconjugatetransposeof W,and S is a diagonal matrix containing singular values of the original matrix. To ensure having a right-handed coordinate system the procedure in equation 3.5 is used. d = sign(det(wv T )) 1 (3.5) Finally, it is possible to calculate the optimal rotation matrix U as shown in equation U = W VT (3.6) 0 0 d Final Translation Kabsch algorithm is used to calculate the optimal rotation between two sets of points but to get the wanted results a translation is also required. A new center of mass for the moving point set is acquired by applying the rotation matrix U from equation 3.6. The translation will be the vector between the center of mass point in the first set and the one in the second, as shown in equation 3.7. T = R b U R a (3.7) where T is the new translation and R b as well as U R a are the center of mass points for the two point sets. 3.2 Manual Point Cloud Alignment In the previous section the user defined the point pairs and thus decided which points should be matched with each other. This implementation will allow the user to mark areas of interest with a point cloud containing an arbitrary number points without specifying pairs, and then perform an alignment procedure. All the alignment steps can be seen in figure Adding Point Clouds The points are for this implementation added by the user by using a point-drawing toolbasedonthemousecursormovement. Asthepointsarenotpairedwithones on other meshes before the alignment phase there is no point in using different colors. In the implementation the colors for all of the sphere representations of points on a single mesh have the same color. The Isolation Mode mentioned in

28 3.2 Manual Point Cloud Alignment 17 (a)initial positions. (b) Isolate and rotate the pieces. (c)addpointcloudsto the pieces. (d) Disable the isolation mode. (e) Perform alignment. Figure 3.3: Manual Point Cloud alignment performed on two pieces of a clinical tibia case. Section 3.1 is also used when adding points in this implementation if there is a need to get a better view of the fractured areas Matching the Point Clouds The main differences between the point cloud matching and the point pair matching is that there is not necessarily the same amount of points on the two meshes and that it is not defined which points that are pairs. After automatically defining the pairs using an algorithm the pieces can be matched using Kabsch algorithm as previously explained in Section As the pairs are not pre-defined the program needs to decide on which points should be aligned with each other. This can be done using the Iterative Closest Point (ICP) method which can be said to consist of the following six steps as described in Szymon et al. [22]: 1. Selecting set of points on one or both meshes. 2. Matching the selected points to samples in the other mesh. 3. Rejecting the point pairs depending on the point sets or the pair itself. 4. Assigning an error metric depending on the point pairs.

29 18 3 Solving 3D Puzzles (a) Closest Point: each point on the moving object (left) finds its closest point on the other mesh and forms a point pair. (b) Mutually Closest Point: a point pair is formed if two points recognize each other as the closest point in the other mesh. Figure 3.4: Two different point selection methods for the ICP algorithm which are tested and compared in this thesis work. 5. Minimizing the error metric value. For the implementations where ICP is used within this report the starting point is the implementation provided by Zhang et al. [27], where a basic version of the algorithm is explained in detail. Some simplifications are made to the algorithm such as the maximum distance is set manually and that when the registration between the point clouds is bad the search distance is reset to the maximum distance instead of the value ξ which is proposed in the report. Another alteration is that a maximum fail counter is included which is incremented whenever the minimum improvement condition is not met. We believe that there is a possibility that a bad matching which does not improve the result may be able to lead to further improvement in a subsequent iteration. Selecting the points for matching can be done in many different ways but in this report two different approaches are used and compared with each other. One is a simple closest point algorithm where each point in the first point set is matched with the closest one in the second as shown in Figure 4.15b. The other is one where points are only used if they are mutually closest to each other. This meansthatapointfromthefirstsetfindstheclosestpointinthesecondsetwhich does not find any other closer points, see Figure 4.15a. The error metric used for evaluation is the average distance between each point in the first point cloud to its closest point in the second set. 3.3 Automatic Point Cloud Alignment This implementation is similar to the previous one, however one big change is that the fracture detection and point placement is carried out automatically.

30 3.3 Automatic Point Cloud Alignment 19 There are also some other features which are deemed necessary to perform an automatic fracture reduction for the three different cases used in this report Procedure Overview Several different methods are combined to get a completely automatic puzzle solver. Some parts are only performed once as an initialization phase and others are used iteratively to end up with the best possible solution. Below is a pseudocode overview for the automatic alignment algorithm. 1. Initialization: (a) Find the fractured areas on all pieces and save the points positions. (b) Filter the point clouds using a distance filter for each neighbor mesh. (c) Calculate the order of matching of the meshes. (d) Improve the initial position between the pieces. 2. Iterativecalculations: (a) Match all pieces using a version of ICP until there is no improvement. (b) Increase the number of points using the existing neighbor points Finding Fractured Areas Previously the user was responsible for adding points on the fractured areas which should be aligned. In this implementation all of the steps in the alignment will be done automatically, which means that there must be an automatic detection of these areas of interest. The algorithm used is based on the one mentioned in Fürnstahl et al. [9]. Each vertex on the mesh has its normal compared to nearby vertices within a radius ˆσ to find an average angle difference, see equation 3.8. If the angle difference is too small it is not accounted for as shown in equation 3.9. Finally, the vertex is said to be on a fractured area if the average normal difference is within [π/2 t1,π/2+t1]. F(x j ) = 1/Nˆσ f (x j,x k ) (3.8) x j ǫ ˆσ,x k x j Where Nˆσ is the number of vertices within the search radius which were not zero in equation 3.9. (n f (x j ) = j,n k ), if (n j,n k ) > t2 (3.9) 0, otherwise In Figure 3.5 a visualization of the result is shown where the red arrows are the normals discarded with respect to the variable t2. If these would also be included the vertex x j in Figure 3.5a would be wrongly classified as a fracturepoint. Intheoriginalreportthethresholdvaluesusedare t1 = π/8and t2 = π/5but as we use different meshes we will perform tests to see which values are optimal.

31 20 3 Solving 3D Puzzles (a) x j FL. (b) x j FL. (c) x j FL. Figure 3.5: Three examples of possible outcomes when using the normal direction deviation fracture line (FL) detection on a point x j [9]. Optimization Performingthefracturedetectiononameshwithalotofverticesrequiresalarge amount of calculations which require time. Because of this two optimizations are added to this method. Firstly, a spatial grid is used for each of the meshes which contains all of the vertices in 3D space cubes. Whenever the program needstocheckforneighborverticeswithinaradiusthegridallowsafastlook-up of nearby vertices. Lastly, to optimize the mesh vertex loop a thread pool is used Classifying Fracture Area Points After the fracture points have been found using the vertex normal method there is a risk of having false points due to noise in meshes which should not be used. Another issue is how to know which points should be used for alignment with each neighboring mesh. To solve both of the problems we suggest the use of a distance filter classifier to define what we call neighbor points. For each of the meshes with points that are classified as belonging to a fractured area, go through each of the neighbor meshes fracture points and find the closest fracture point in the current mesh, see Figure 3.6. These neighbor points will be used when performing an alignment with that specific object. As the bone pieces are assumed to have a good initial alignment the closest distance is deemed to be a good classifier. Only the closest points will be used and thus the false positives will be removed unless they are closer than the real fractured area which is unlikely if the pieces are not considerably displaced Hierarchical Matching Order When it comes to matching of several pieces the order of matching may become an issue. A simple solution by having all of the pieces match each other each iteration may result in nothing happening at all and a large piece matching with a smaller piece may an already good matching result. The order of matching proposed in this implementation is the following:

32 3.3 Automatic Point Cloud Alignment 21 (a) Points found using the fracture detection. (b) Distance filter on left mesh. (c) Distance filter on right mesh. (d) Classified neighbor points. Figure 3.6: Point classification using a distance filter which for each fracture point belonging to the current mesh finds the closest fracture point of a nearby object. This closest point is classified as a neighbor point to the current mesh. 1. Select a bone which is static and put it first in the list of objects. 2. Sort the other objects depending on the number of points used for matching with the static object. All meshes which are not static are then aligned with the ones above them in the queue. If an object aligns only with static meshes and does not improve its position it becomes static as well. Repeat until there are no non-static objects left Improve Initial Conditions When the first search for neighbor points has been performed there is a risk that the points are not optimally placed because of errors in the initial alignment between pieces. As ICP is known for getting stuck in local minima this may affect the end result and have the objects reach a solution which is not optimal. To solve this problem two types of approaches for improving the initial position are suggested and will be compared in how they solve the three different cases: Overlap and Match The idea is that two already overlapping objects can get a better alignment using the neighbor points compared to displaced ones. Creating the overlap is done by spreading the points found from the classification to vertices within a radius and then aligning the objects. After the original points are used for the alignment with the new start position to move the overlapping objects apart, see Figure 3.7. Move Along Plane/Line In this case what is done is to improve the initial position without creating an overlap to be able to improve the points using the distance filter before matching. A middle-point is calculated using each object s neighbor points and the

33 22 3 Solving 3D Puzzles (a) Original neighbor points. (b) Spread points to nearby vertices. (c) Align to create overlap. (d) Overlap, original points. (e) Align using original points. Figure 3.7: A way of improving initial position between meshes by spreading points to nearby vertices and align the new points to create an overlap. The overlapping position is used for the alignment using the original points.

34 3.3 Automatic Point Cloud Alignment 23 (a) Calculate middle-point for each mesh (yellow). (b) Project middle-point difference on a line to get translation. Figure 3.8: Improving the initial position between two meshes by translating the moving mesh the difference in distance between the objects middlepoints. This is done along a line from the middle-point to the furthest point in the same mesh to avoid overlap. displaced mesh is moved the difference between these points. To avoid overlap the translation vector is projected on a line/plane which direction is not towards the other object. This can be done using the Singular Value Decomposition from Kabsch algorithm which contains an approximated plane from a point set. Another suggestion is to use a method which uses the middle-point and creates a line from this point to the furthest point, see Figure Increase Neighbor Points As shown in Figure 3.9b a match of the original points is not enough for a good match between the pieces. This is due to the simplicity of the point classification method in Section To improve the result for cases with more curvature a possibility to iteratively increase the number of points is suggested. After an alignment iteration has been performed a new point classification is performed. In this case each neighbor point on one mesh is compared to the fracture area points on the other mesh to see which one is the closest. If the closest point is already classified nothing happens, and if not it will be included for the next point cloud matching. In Figure 3.9 it is illustrated how this method can improve cases when the original points are not enough for a good alignment Generating Bone Meshes The surface extraction is done using the marching cubes algorithm as described by Lorensen et al. [18] and the meshes are simplified using Meshlab s Quadric Edge Collapse decimation. After the decimation it is possible that vertices have been removed on smaller surfaces, which are often fracture areas. As the fracture

35 24 3 Solving 3D Puzzles (a) Starting position with neighbor points. (b) Align using the original points. (c) Update nearest points (left). (d) Update nearest points(right). (e) Increased points on both meshes. (f) Align using the additional points. Figure 3.9: Two iterations of ICP alignment on two pieces. After the first alignment more neighbor points are added using a point distance classification between current neighbor points and fracture points. Lines between points indicate that new neighbor points are added.

36 3.3 Automatic Point Cloud Alignment 25 detection algorithm requires vertices to function each edge of the mesh is split once after the decimation is performed.

37

38 4 Results Within this chapter the results from the different implementation and method tests are presented, as well as the three different cases used for evaluation. Each testwasrunonacomputerwithanintelcorei7-3612qm2.1ghzquadcoreprocessor, a NVIDIA GeForce GTX 640M 2GB GDDR5 video card, and 8GB DDR3 memory. 4.1 A Quantitative Implementation Comparison Five pieces of a broken hollow egg shell model with a maximum height of 47 millimeters and width of 35 millimeters was used for a quantitative evaluation, seefigure4.1and4.2. Thetotalnumberofverticesforthiscasewas Four ofthepiecesweredisplacedasshowninfigure4.3. Thegoalofthetestistoalign them with the static bottom piece and compare their positions and rotations to the originals to calculate the alignment error. This is calculated by using the root mean squared deviation (RMSD) method. Another recorded aspect is the interaction time required and the calculation time, when applicable. This test was performed five times for each implementation. Manual Point Pair Tests using Manual Point Pair is shown in Table 4.1 and had the lowest mean distance and rotation error when compared to the other implementations. The tests results varied and MPP had the largest observed rotation deviation when compared to the mean value. The interaction time required varied from 110 to 215 seconds but does not correlate with the RMSD result. It indicates that the first try was good enough in some tests and that others required redoing one or several alignments. An image of the result from test number five is shown in 27

39 28 4 Results Figure 4.1: A shattered egg shell model created in Blender. (a) Front view. (b) Side view. (c) Back view. Figure 4.2: Five pieces from the egg shell model which are used as a golden standard for the qualitative tests. (a) Front view. (b) Side view. (c) Back view. Figure 4.3: Five pieces from the egg shell model after applying manual rotation and translation displacement.

40 4.1 A Quantitative Implementation Comparison 29 Table 4.1: Manual Point Pair tests results. Measure Distance RMSD (millimeters) Rotation RMSD (degrees) Interaction time (seconds) Test 1 Test 2 Test 3 Test 4 Test 5 Mean (a) Front view. (b) Side view. (c) Back view. Figure4.4: Test 5 in Table 4.1 for Manual Point Pair, egg shell case. Figure 4.4. Manual Point Cloud The Manual Point Cloud implementation got the largest distance deviation comparedtoitsmeanvalueandthehighestmeanerrorwhenitcomestodistance,as seen in Table 4.2. Interaction time required had less variation than for the MPP implementation but a slightly larger average time. Both manual methods show a variety in result values each test which depends on how well the user places the points. In Figure 4.5 test case number one is presented. Automatic Point Cloud Finally, the Automatic Point Cloud implementation, as shown in Table 4.3, had the lowest interaction time required and the total time was approximately half of the manual implementations. APC had the lowest deviation when comparing each test s distance and rotation error with the mean values, which means that theresultsweresimilareverytimethemethodwasused. Ithadthelargestmean rotation error out of the three methods and when it comes to distance it was worse than MPP but better than MPC. Figure 4.6 shows the visual result from test number two.

41 30 4 Results Table 4.2: Manual Point Cloud test results. Measure Distance RMSD (millimeters) Rotation RMSD (degrees) Interaction time (seconds) Test 1 Test 2 Test 3 Test 4 Test 5 Mean (a) Front view. (b) Side view. (c) Back view. Figure4.5: Test 1 in Table 4.2 for Manual Point Cloud, egg shell case. Table 4.3: Automatic Point Cloud test results Measure Distance RMSD (millimeters) Rotation RMSD (degrees) Interaction time (seconds) Calculation time (seconds) Test 1 Test 2 Test 3 Test 4 Test 5 Mean (a) Front view. (b) Side view. (c) Back view. Figure4.6: Test 2 in Table 4.3 for Automatic Point Cloud, egg shell case.

42 4.2 A Qualitative Implementation Comparison 31 (a) Front view. (b) Back view. Figure 4.7: A clinical pelvic case used for the qualitative evaluation with two pieces to align. (a) view. Front (b) Top view. (c) Left view. (d) view. Right Figure 4.8: A clinical tibia case used for the qualitative evaluation with four pieces to align. 4.2 A Qualitative Implementation Comparison An implementation test was performed on two real clinical cases, one pelvis which has two broken pieces with a vertex total of and one tibia with four pieces and a total of vertices, as shown in Figure 4.7 and 4.8. Unlike the egg shell model these cases have no known truth and therefore it was decided to use a qualitative evaluation considering visual result, time spent, and user satisfaction. Two test users participated in the qualitative tests, one which is a clinical expert and has previously used Sectra s 3D pre-operative planning tool. The second user has no prior experience in orthopaedics but is familiar with computer graphics and tools which require 3D interaction.

43 32 4 Results (a)user test 1. (b)user test 2. Figure 4.9: Manual Point Pair user test results, pelvis. (a)user 1 test. (b)user 2 test. Figure 4.10: Manual Point Cloud user test results, pelvis. (a)user 1 test. (b)user 2 test. Figure 4.11: Automatic Point Cloud user test results, pelvis. The dark spots represent the vertices found in the fracture detection.

44 4.2 A Qualitative Implementation Comparison Pelvic Case The users managed to solve the pelvic case using the two manual methods as well as using the automatic one and provided feedback on both the manual and automatic methods which is presented below. Manual Alignment BothoftheusersfoundtheManualPointPair(MPP)andtheManualPointCloud (MPC) methods useful for the pelvic case. They valued that they were in control and that it was quite easy to reach a satisfactory result. The first test user got a better result using the MPP method as shown in Figure 4.9a and the second preferredmpc,seefigure 4.10b. Inbothcases theymanaged toreacharesultin a few minutes time. The clinical expert mentioned that the lack of mesh detail madeitdifficulttoseewheretoplacethepoints,whichmeansthatthereisanegative aspect to the mesh reduction when it comes to manual approaches. One of theusersmisplacedapointusingthemppmethodthefirsttimeandhadtoredo the procedure. This however this did not negatively affect the user satisfaction as the person felt responsible for the error. Automatic Alignment When using the automatic approach none of the users were as satisfied with the result compared to the manual methods, see Figure Even if the automatic result was decent they both managed to get a better looking result using manual methods. Boththoughtthatitwaseasiertousetheautomaticmethodandittook less time, however as the result was not good enough and the manual methods were not too tedious to use they were preferred. They commented on the lack of feedback during the automatic calculations as well and wanted to see a progress bar or something similar during the waiting time of approximately twelve seconds. Both users commented on the negative aspect of not being in control, but the clinical expert said that it would be okay as long as the result is good Tibia Case Bothofthetestersattemptedtoreducethefractureforthetibiacaseusingeachof the implementation methods, however none of them succeeded using the manual tools. The reasons for this are the same as mentioned for the pelvic case, but as thiscasehasmoreflatsurfacesitwasevenmoredifficulttoseewheretoplacethe points. With more pieces it was also confusing to use the isolation mode which moves the bones from their original positions, especially when several pieces are interacted with at the same time which is necessary for the MPP implementation. Another issue was to know in which order to align the pieces, which was especially difficult for the test user without prior medical knowledge. The clinical expert mentioned that the case used was incomplete and missing a bone which normally is important when solving this type of fracture.

45 34 4 Results (a) view. Front (b) Top view. (c) Right view. (d) Left view. Figure 4.12: Tibia case reduced using the Manual Point Pair implementation. As there were no results from these tests it was decided to present our own results when performing the fracture reduction using each implementation, as well as some observed pros and cons for each of them. Manual Point Pair We found this method to be tedious to use for the tibia case. It is difficult to see exactly where to place the points even when knowing how the pieces should fit together. Using the isolation mode on several pieces at the same time can be confusing if the case has not been solved before. Because of bad point placement we had to redo the fracture reduction several times until we were satisfied with the final results. In about eight minutes it was possible to get a solution which is shown in Figure Manual Point Cloud This was the method which got the best results for this case. As we knew which areas should be aligned with each other and the case is about matching thin surface lines it was easy to perform a fracture reduction. Another positive aspect is that unlike the MPP method it is not necessary to apply every second point to another mesh. Here it was possible to add all points necessary for the alignment to a single mesh and then have it return to its original position which was easier than when isolating all pieces at the same time. Using this method it was possible to reduce the fractures in a good way in four minutes without any errors, as shown in Figure Automatic Point Cloud Using this method was the easiest as the user only needs to select which pieces should be aligned with each other. Depending on which piece was selected as static the result will be different, but as we knew a good order of alignment this was not a problem. Calculation time was approximately one and a half minutes to get the results shown in Figure As mentioned by the test users the lack

46 4.3 Algorithm and Parameter Tests 35 (a) view. Front (b) Top view. (c) view. Right (d) Left view. Figure 4.13: Tibia case reduced using the Manual Point Cloud implementation. (a) view. Front (b) Top view. (c) view. Right (d) Left view. Figure 4.14: Tibia case reduced using the Automatic Point Cloud implementation. of feedback along with the long calculation time can be frustrating but can be solved by doing other things while waiting. 4.3 Algorithm and Parameter Tests As mentioned in Chapter 3 several different approaches and parameter values have been evaluated to improve the result of the implementations. Within this section the final configurations are presented as well as why they were selected Iterative Closest Point As described in Chapter 3 the ICP alignment method has several parameters which need to be set. Different combinations were tried to get the best alignment result possible. In Table 4.4 the parameter values used for the rough alignment and the final adjustment iterations are presented.

47 36 4 Results Table 4.4: Parameter values used for ICP alignments. Parameter Rough Alignment Final Adjustments Maximum angle (degrees) Maximum distance 20 5 Fails allowed Minimum improvement (a) Mutually Closest Point. (b) Closest Point. Figure 4.15: A comparison of two point selection methods when solving the egg shell case, where (a) required less than half of the computation time compared to (b). Fracture areas are of black color, and cyan and magenta denotes neighbor points from the point classification or neighbor point increase methods. Point Selection When using the ICP algorithm two approaches for selecting points for the matching was tested. Matching each of the points with its closest points in the other mesh yielded good results, but we got similar visual results using the mutual match method. We decided on the latter method as less points were used for the matching and thus the calculation time was lower. Figure 4.15 shows a comparison of the two methods using the same parameters where the mutually closest point required less than half of the other method s calculation time Detect and Classify Fracture Points A large part of the thesis has been to find good parameters for the fracture detection which would work both for a premade egg shell case and for simplified meshes of real clinical cases. An issue has been that the meshes which were constructed using the marching cubes algorithm and then simplified contained a lot of noise. This affected the normal directions when looking at a small radius but was possible to ignore for a larger radius. Though when using a radius which is too large you may also lose interesting local edges. Another problem was that the meshes have different curvature, some are for example more round while others are straight, which makes it more difficult to select the values for t1 and t2. To compensate for this different combinations

48 4.3 Algorithm and Parameter Tests 37 (a) Fracture points. (b) Neighbor points. Figure 4.16: Fracture points (dark colors) are detected and classified as neighbor points. The colors of the points indicate which mesh they are neighbors to and will be used to align with. were tried until all three cases had a minimal amount of false positives without missing out on any fractured areas using the same parameter values. Variable Values It was decided not to use the original configurations for the fracture detection as they did not yield a good result for the meshes used [9]. By using the values t1 = π/10,t2 = π/3and ˆσ = 5thenumberoffalsepositiveswereminimalandall of the fractured areas were found. The search radius needed to be slightly larger than the thickness of the bone/egg. Figure 4.16a shows the detected fractures as dark color where it is evident that the amount of false positives varies depending on the meshes. In Figure 4.16b the points are classified and most of the false positives found have been removed. The classified points have the same color as the mesh they are supposed to be matched with Improve Initial Position After testing the automatic method without using an initial position improver it wasclearthatboththeeggshellcaseandthetibiacasewerenotinneedofinitial improvement for a satisfactory result. The pelvic case on the other hand could gain from having its initial position improved. Overlap and Match Theoverlapmethodwasusefulforallcasesbuttheradiushadtobedifferentfor each case as the topology and size of pieces differed. With the wrong radius the final result got worse when using this method and therefore it was decided not to use it.

49 38 4 Results Figure 4.17: Singular Value Decomposition plane approximation, pelvic case. Theredlineistheapproximatedplane snormalandthegrayandwhite ones are vectors on the plane. Move Along Plane/Line Findingaplane/linetomoveameshalongprovedtobedifficultforageneralcase as it is dependent on the quality of points found during the classification. Using the Singular Value Decomposition from Kabsch algorithm it was possible to find a plane for a point cloud for some cases, as shown in Figure 4.17, where the red lineistheplanenormalandthegrayandwhiteonesbelongtotheapproximated plane. However the method was not reliable and sometimes found planes which were incorrect and therefore it was decided not to use this method. The method creating a line from the middle-point to the furthest point worked well for the pelvic case but as the movement was too small it was decided to use it several times. Between each iteration more points were added using the point classification until there was no longer any improvement. This worked well for the egg case and pelvic case but when performing it on the tibia case which already has a good initial alignment it created a slight overlap between some pieces. The idea to classify new points using the improved position was not used as the point classifier does not work well for overlapping meshes. By using the original points it was still possible to improve the pelvic case result slightly while it did not negatively affect the other two cases, see Figure Increase Neighbor Points As there is a problem when the initial points are not perfect because of the simplicity of the classification method some of the cases were improved by using the increase points method. In Figure 4.19 a comparison between a solution using this method and one without is shown for the egg shell case. In this case it is clear that by increasing points it is possible to help solving cases where the initial points are not good enough.

50 4.3 Algorithm and Parameter Tests 39 (a) Alignment without initial improvement. (b) Alignment using initial improvement. Figure 4.18: A comparison of using Automatic Point Cloud alignment for the pelvic case without any initial position improver and when using the move along line approach. (a) Alignment without increasing neighbor points. (b) Alignment when increasing neighbor points. Figure 4.19: A comparison of Automatic Point Cloud alignment for the egg shell case while iteratively increasing the number of neighbor points and when not using this approach.

51

52 5 Discussion In this chapter the different methods, their tests and results will be discussed. Also suggestions for future work will be brought up. 5.1 Implementation Discussion This thesis is about comparing three methods with varying amount of required user interaction for aligning bone meshes with each other to solve a 3D puzzle. In this section the pros and cons will be discussed and also if it can be said that manual or automatic solutions are superior Manual Point Pair The method relies heavily on correct user input and will provide a good result if the points are placed correctly, regardless of the initial positioning of the bone pieces. It is good for users which want to be in complete control of how the pieces will be aligned. According to the quantitative results this implementation had the best mean results for both the rotation and translation error, as well as thesmallestobservederror. Ontheotherhandtheresultsvariedfromonetestto anotheralthoughthesamecasewassolvedmanytimesbeforeandtheuserknew where to place points for a good result. It shows that it may be difficult to know the exact position for points even if the case has been solved before, which can bebecauseoflackofdetailsinmeshes. Insuchcasesthismethodwhichrequires specific point pair input may be disadvantageous to use. Another negative aspect is that every other point needs to be put on a target mesh (one which is not moving). This means that the user needs to interact with several mesh pieces at the same time when aligning. As the bones usually are close to each other it is necessary to have tools to be able to view fractured areas 41

53 42 5 Discussion of several pieces at the same time which was the idea with the Isolation Mode. Accordingtothetestusersthistoolcouldbeconfusingandwasnotenoughtobe abletogetagoodoverview,especiallywheninneedofaclose-uporaligningone mesh with several other bones. During both tests points were placed completely wrongbecauseitwasdifficulttovisualizehowthepieceswouldbealignedinthe Isolation Mode. An interesting aspect however is that when the user is in charge of the alignment, failure is more likely to be blamed on the user s inexperience than the tools of the system Manual Point Cloud This method can be useful when there is not enough detail on the bones to know exactly where to place points, for example with large flat surfaces to align without any,orwithfewlandmarks. Thetestuserswereabletosolvethepelviccaseusing thismethodandtheclinicalexpertfounditeasiertousethismethodforthetibia casethanmpp.exceptthatthepieceshadfewobviouslandmarksthereisalsothe positive aspect that you do not need to specifically define pairs of points. If the user can spot fractured areas on each mesh that should be aligned and fill them with many points the ICP algorithm will provide an alignment which considers the meshes topology to match as many points as possible. As the evaluation is based all of the points the method is more robust when it comes to bad points compared to the MPP method, as long as a majority are correctly placed. A negative aspect of this method is that the initial position and rotation of eachmeshisimportantasitistheclosestpointswhicharealignedeachiteration and that poor initial alignment will likely lead to a bad result. Another is that the algorithm will try to find the best matching between the point sets and stop if there is no improvement in a few iterations. As it is right now there is a risk of getting stuck in a local minimum where there is still room for improvement but itisnotfound. Anexampleisiftherearemanypointsononesurfaceandlesson another the areas with many points will be prioritized as the matching score is improved if most of those points have good alignments. In other words the user may place points correctly but still get unsatisfying results Automatic Point Cloud In all of the cases the automatic solution is the fastest as well as requires the least interaction and knowledge from a user. For the cases with several pieces with a good initial alignment and thin contact surfaces this method works well, like for the egg Shell and tibia case. During the tests with the pelvic case of two pieces with round displaced contact surfaces it was not as successful as the manual methods. When comparing the calculation time for the three cases it can be seen that it does not only depend on the total vertex count of the meshes. A large amount of vertices makes it possible to detect more fracture points. The number of fracture points will then affect the amount of neighbor points, which are used for several of the automatic method s algorithms. For each iteration all of the moving pieces

54 5.2 Evaluation Drawbacks 43 are matched with the pieces higher up in the hierarchy, which means that the number of meshes also may affect the calculation time. Other aspects, such as the fracture case and how well pieces are initially aligned, may also be of importance. The fracture detection algorithm is sensitive to variable values and has a risk of finding false positives because of local curvature or noise in meshes. The point classification removes these points but unfortunately it can also remove ones which should be used for matching, especially for displaced or non-flat meshes. An issue with the APC implementation is that it is sensitive to mesh placement and topology. As there is no good evaluation of the fracture reduction the final result depends on the initial points as well as the points generated during the alignment process. Another problem is that there is no visual feedback provided during the calculation phase, which means that the users do not know how long they have to wait. As the automatic method can take a few minutes to perform for several pieces the user may become irritated, especially if the result after waiting is not good enough and they have to perform additional manual adjustments. We have alsonoticedthatthemethodhasissueswhenthepuzzleismissingpiecesasthere is a risk that bone fragments are used to fill the holes which is not necessarily a correct alignment. The initial position improver was useful in one out of three cases, which means that its calculation time is sometimes unnecessary. Using the point improvermayimprovetheresultsasshowninfigure4.19butmayalsobeunnecessary in some cases and thus only increase the processing time Manual VS Automatic Itisnotyetpossibletosaythatoneofthemethodsismoreusefulthananotherin general, nor if manual or automatic ones are superior. An example of this can be seen in the quantitative tests where each implementation s best result was better than the worst result of the other implementations. One thing which is clear is that the automatic solution can save much time for an orthopaedist, but as the application is for pre-operative planning the quality of the fracture reduction is more important than the time spent. If it cannot be guaranteed that the automatic solutionsaregoodenoughtheremustbeapossibilitytousemanualtoolsaswell. 5.2 Evaluation Drawbacks In Chapter 4 a qualitative and quantitative approach for evaluating and comparing the implementations was presented. Because of lack of time there are several aspects in which the evaluation methods are lacking Quantitative Evaluation The quantitative evaluation method with the egg shell was used to be able to get numbers on how well the different algorithms worked, but there are several drawbacks to this method. One of the more obvious problems is that it shows

55 44 5 Discussion how well the methods work for an egg shell created in a 3D modeling program and not a real bone mesh. A better idea would have been to create a bone model and breaking it to get a golden standard for a more realistic case. Another issue about this evaluation is that it is the root mean squared of the distance and rotation which is measured which is not necessarily a good error measure for clinical purposes. Finally, we who have created the model and practiced solving it many times performed the tests. This is advantageous for the manual approaches as the test user knew where to place the points for the best results from previous experiences. It also increases the risk that the tests were not impartially performed Qualitative Evaluation As the quantitative evaluation could not be related to the real cases which the methods were used on the qualitative evaluation was also added, however more time would have been necessary to perform a good evaluation. One of the drawbacks is that there were only two test users which limits the generalizability of the conclusions. Another issue is that the users had no prior experience with this system and that only one of them had knowledge within orthopaedics and pre-operative planning. This means that the comparison becomes unfair as the manual tools are difficult to use and the process of solving 3D bone puzzles is new. An example is that they both failed at solving the tibia case using the manual tools provided. The ideal testers would have been orthopaedists which are already familiar with the tools of the system and try to solve a new clinical case using the tools provided. Meshes that were used for the tests were highly optimized and therefore lacked some clinically important details which was a problem for the users. A larger variety of fracture cases could have been used to compare how well the different methods fare against each other to see which are optimally used for certain fracture types. Especially as the tests have shown that different methods were useful for the two cases used. 5.3 Future Work As shown in Chapter 4 the three implementations used in this thesis each yielded promising results. However, the methods can still be improved in several ways, which will be discussed within this section Planning Analysis One of the issues which appeared during this thesis was how to evaluate how well the fracture is reduced. If a reliable method for automatic evaluation of the fracture reduction would exist it would help with a fair comparison of the manual and automatic methods. It could also improve the automatic approach asthealgorithmcouldknowwhenithasreachedagoodresultwithoutthevisual feedback from a clinical expert.

56 5.3 Future Work Calculation Optimization There are several ideas on how to improve the time spent on calculation when it comes to the automatic solution. Threading and Data Structures As of now a spatial grid is used, though only for fracture detection and finding newpointsonfracturedareas. Athreadpoolisusedforthefracturedetectionas well but otherwise everything is run on the same thread. One of the most time consuming tasks, the ICP algorithm, is currently not using any optimization. By including a faster data structure eg. spatial grids, some threading, and by tweaking the procedure parameters we believe that calculation time can be significantly lower than it is right now. Automatic Tool Selection Several tools were added for the automatic implementation to be able to get a goodsolutionforallthreecaseswithoutanyuserinterference,butnotallofthem were necessary for each case. To avoid unnecessary computations an automatic selection of tools for the different fracture cases would be useful. Pre-Processing Some of the calculation steps mentioned in this report takes significant time to perform and therefore it would be preferred if they could be a pre-processing step rather than something which happens in real-time with the user present. As the method described is relying on meshes to find the fractured areas and match these together it is relevant to have as much data represented as meshes instead of volume data. To be able to efficiently create meshes out of all bones in the data set it would be preferred if both the segmentation and mesh reconstruction was performed automatically, for example using the methods described by Thomas et al. [24] Fracture Detection using Face Normals As it is now the fracture detection algorithm is calculating the difference sum between each vertex normal direction and those of its neighbors. One of the issues using this approach is that there must be vertices on the fractured areas, otherwise they cannot be found. This is a problem as mesh reduction algorithms such as Quadric Edge Collapse Decimation tend to remove vertices on small surfaces. This problem was solved by using subdivision where each edge is split on its mid-point after the mesh reduction. To avoid this work-around we suggest that an adjustment can be made to the algorithm to compare face normals instead of vertex normals.

57 46 5 Discussion Improved User Interaction During this thesis the focus has been more on the comparison of the matching methodsandlessonimprovingtheuserinteraction. Toallowtheusertozoomin on bones and easily interact with several pieces at the same time without losing the understandability of the clinical problem would be a great improvement, for example by using several views windows. Right now there is no possibility to edit or correct the points if they are placed badly or result in a bad alignment which is also a feature that would improve the manual approaches ICP Configuration Tests Within this thesis there were some comparisons when it comes to parameter values and which closest point methods and evaluations should be used for the best result. As mentioned in the Background there are many different aspects to consider when matching using an ICP algorithm. A comparison between the chosen configuration presented in this thesis and other available options would help reaching a conclusion on how an optimal solution would work Point Match Weighing When using the ICP algorithm there may be matches better than others because of initial alignment errors. To increase the significance of the points which are good matches would be an improvement to the implementation. Normal Direction Ifthevertexononemeshmatcheswithoneonanothermeshwhichhasanormal direction which is oppositely directed there is a lower risk of a false match. An ideaistousethistoapplysmallerweightsto,orremovepointpairswhichshould not be matched with each other, see Figure Final Adjustments As is previously discussed there are times where each of the methods will not yield a satisfactory result. For example pieces could end up being displaced, too far apart, or overlapping with each other. Adding a possibility to improve the result of matching, either using manual or automatic approaches could improve the final result Visual Feedback During calculations users get no information regarding how long it will take or what is happening. A progress bar or an animation which shows the iterative steps could be used to provide the users with visual feedback.

58 5.3 Future Work 47 Figure 5.1: Vertex normal-based weighing which considers if the point pairs have normals of opposite directions. If they do (green) they should have more influence on the result than the ones which do not (yellow) [19] Transparency Whenthereareseveralobjectsalignedwitheachotheritmaybedifficulttoseeif there is any overlap between bone meshes after the matching process. This means that the evaluation of the solution given by manually or automatically aligning piecescanbeadifficulttaskbyitself. Tomakethepotentialoverlapmorevisible to the user we suggest that transparency is used, similarly as in Fürnstahl et al. see Figure 5.2 [9] Match Bones Using Bone Models In several papers regarding orthopaedic pre-operative planning, for example Fürnstahl et al. [9] and Okada et al. [20], a method using the mirrored contralateral bone as a bone model which the fractured bone can be aligned to. In the cases when the initial positions of the pieces are bad this may improve the final result when using methods such as ICP. This could possibly replace the current initial position improver Multiple Mesh Resolution When finding the fractured areas of the meshes each vertex normal is compared the ones that are nearby. This means that when handling a large amount of vertices there will be many comparisons and thus a lot of calculations. An idea for future work is that a lower resolution mesh can be used when finding the areas of interest and that the points found can be mapped onto the original mesh. Another approach could be to find the mesh transform for the alignment using low resolution models and then applying it to the high resolution mesh.

59 48 5 Discussion Figure 5.2: Transparent bone meshes with colored fracture lines to make it easier to identify overlapping objects [9].