Three-dimensional Reconstruction System for Automatic Recognition of Nasal Vestibule and Nasal Septum in CT Images

Journal of Medical and Biological Engineering, 34(6): 574-580 574 Three-dimensional Reconstruction System for Automatic Recognition of Nasal Vestibule and Nasal Septum in CT Images Chung-Feng Jeffrey Kuo 1, * Yueng-Hsiang Chu 2, * Chin-Liang Liu 3 Fang Tzu Yeh 1 Han-Cheng Wu 1 Wei Chu 4 1 Graduate Institute of Automation and Control, National Taiwan University of Science and Technology, Taipei 106, Taiwan, ROC 2 Department of Otolaryngology-Head and Neck Surgery, Tri-Service General Hospital, National Defense Medical Center, Taipei 114, ROC 3 Neural Surgeon Department, Taipei City Hospital, Taipei 106, Taiwan, ROC 4 Department of Materials Science and Engineering, National Taiwan University of Science and Technology, Taipei 106, Taiwan, ROC Received 31 May 2013; Accepted 2 Oct 2013; doi: 10.5405/jmbe.1583 Abstract At present, computed tomography (CT) images of interest for medical diagnosis are manually selected among a series of images by physicians. This study uses image processing to automatically select the regions of the nasal vestibule and nasal septum using a back-propagation neural network. Three-dimensional (3D) image reconstructions of the nasal regions are conducted to obtain the 3D spatial information. 3D images of the nasal cavity, nasopharynx, and paranasal sinuses are also reconstructed. The representative s are labeled in three surgically risky locations, namely the anterior cranial fossa of the brain and the inferior and medial edges of the orbital rim. The nasal and paranasal sinus conformation and marked s are presented to assist surgeons in the preoperative evaluation. The overall recognition rate of the system for the nasal vestibule and nasal septum is 99.7%. The 3D images are considered to be beneficial by an otolaryngologist from Tri-Service General Hospital in Taiwan. The proposed method can facilitate the preoperative evaluation and improve the quality of medical care. Keywords: Image processing, Back-propagation neural network, three-dimensional image reconstruction, computed tomography 1. Introduction At present, images used in otorhinolaryngology (ear, nose, and throat) are two-dimensional (2D). Physician have to rely on experience to determine the disease type, lesion type, size, and shape. The present study converts a series of 2D images into three-dimensional (3D) images using mathematical calculations and image scientific visualization technology [1]. Using computer image processing, the generated data are presented as an intuitive 3D image. Currently, a 3D reconstruction cannot be automatically processed by a computer, and manual processing is timeconsuming. This study uses MATLAB (The MathWorks, * Corresponding author: Chung-Feng Jeffrey Kuo Tel: +886-2-27908821 E-mail: jeffreykuo@mail.ntust.edu.tw * Corresponding author: Yueng-Hsiang Chu Tel: +886-2-87927192 E-mail: chuyuenghsiang@gmail.com Natick, MA) to develop a system that can automatically select computed tomography (CT) images of the nasal vestibule and nasal septum by integrating image processing and neural networks. Avizo 7.0 (FEI, Burlington, Vermont) is applied for 3D reconstruction. Using 3D images of the skull, nasal cavity, nasopharynx, and paranasal sinuses, an image with detailed nasal anatomy is constructed. The distances between the nasal vestibule and representative surgically risky locations are measured. The 3D image can be used by physician for preoperative evaluation and surgical approach planning. Wang et al. [2] applied scientific visualization technology in the preoperative evaluation of donor kidney blood vessels for an in vivo kidney transplant. They used Amira (TGS, San Diego, Calif) to process the CT images of part of the kidney for 3D reconstruction. Biedert et al. [3] applied Amira to the 3D magnetic resonance imaging (MRI) image reconstruction of a kneecap. The shape, volume, bone shape, and cartilage could be accurately reconstructed from MRI images to assist orthopedic surgery. Wang et al. [4] applied Amira to the reconstruction of 3D MRI image models to demonstrate the entire penile surgical

575 J. Med. Biol. Eng., Vol. 34 No. 6 2014 procedure by integrating animation simulation. Pham et al. [5] used MIMICS software for 3D CT image reconstruction and simulation of a virtual osteotomy. The original and restored virtual data were inputted into the virtual surgical navigation system to guide the surgeon in facial bone structure reconstruction in dental surgery. Zubair et al. [6] provided an overview of the limitations of the computational fluid dynamics (CFD) modeling of nasal airflow. The constraints associated with CFD modeling were reviewed and future studies that could be carried out were discussed. The above study recommended a standardization of the computational modeling procedure, which was essential for studying airflow in the nasal cavity. Haralambidis et al. [7] used MIMICS software for the reconstruction of 3D CT images to present the morphological changes of the nasal cavity caused by rapid maxillary expansion (RME), which was a treatment for obstructive sleep apnea in children, and to calculate the volumes before and after nasal cavity expansion. According to images before and after surgery, the size of the nasal cavity expansion and the status of air flow in the nasal cavity could be determined. Görgülü et al. [8] assessed the impact of nasal cavity RME on the volume of the nasal cavity, conducted image segmentation processing of 3D CT images of the nasal cavity and teeth in the maxillary sinus, and recorded the treatment process information. Matteo et al. [9] conducted the CT image reconstruction of the midline skull base for surface segmentation. The selection of the bone threshold values in the reconstruction process using Amira reconstruction was semi-automatic, and the threshold values of the nasal cavity and sinus were manually selected for reconstruction. Finally, the image after reconstruction was used for the simulation of the volume of interest (VOI). Images after such processing steps were relatively precise and thus used to improve surgical methods in endoscopic surgery. Regarding three patterns of nasal cavity airway flow models, Zhu et al. [10] evaluated the nasal airway flow pattern by using computational fluid dynamics simulation and found similar nasal cavity characteristics for the anterior nasal airway, nasal indices, and nostril shape. By using MIMICS software to analyze the CT images of anterior nasal airway, nasal indices and nostril shapes. The nasal cavity reconstruction of 3D image models could be obtained. Then, the unnecessary parts were eliminated by manual modification and segmentation to make the surface of the nasal cavity model closer to real nasal cavity structure to be highlighted by the user. Surgery simulations of the nasal cavity have considerably important in the surgery planning. In the above studies, most nasal cavity reconstruction methods relied on manual selection of the threshold value and manual segmentation for 3D reconstruction. Since each patient has a large amount of CT image data, manual processing is time-consuming and the preoperative planning of the surgical approach is somewhat difficult. The study thus developed an automatic selection system for nasal vestibule and nasal septum recognition to reduce processing time and provided more detailed preoperative image information and facilitated disease diagnosis and treatment. For 3D image reconstruction, this study employs the marching cubes algorithm [11]. The reconstruction of a 3D volume model for a specific grayscale iso-surface captures the iso-surface based on the pixel value of each image before reconstructing voxels to obtain the 3D image data. Guyomarc h et al. [12] employed the marching cubes algorithm of Amira software and the half-maximum height protocol in three dimensions (HMH 3D) algorithm of Treatment and Increased Vision for Medical Imaging (TIVMI) software for skull reconstruction, and found that the reconstruction results may not be consistent with the original patterns. The present study employs Avizo/Amira software to conduct the nose structure 3D reconstruction, surgical simulation and measurement. 2. Methods A flow chart of this study was shown in Fig. 1. Image processing was used to the CT image to automatically select the regions of the nasal vestibule and nasal septum in CT images using a back-propagation neural network (BPNN). A total of 385 DICOM images with a resolution of 512 512 pixels, a voxel size of 1 mm, a slice thickness of 1 mm, and distances of per voxel of 0.48828125 mm were used. The image processing included four steps: Figure 1. Automatic recognition of nasal vestibule and nasal septum flow chart. Step 1: Head CT image information signal waveform analysis and classify the image data into brain, eyes, nose, mouth and jaw using the BPNN as the classifier. Step 2: Automatically search for the nasal vestibule location information. Step 3: Automatically search for the nasal septum location information. The detailed nasal anatomy was completed using two steps: Step 1: Reconstruct 3D images of the skull, nasal cavity, nasopharynx, and sinus by integrating the nasal vestibule and nasal septum information s. Step 2: Measure the distances from the nasal vestibule and nasal septum to representative s in the three major risky locations.

Three-dimensional Reconstruction of Nasal Regions 576 2.1 Classifier This study classified CT the images into the jaw, mouth, nose, eye, and brain components, and analyzed the CT signal waveforms of the five parts as the characteristic values of the classifier. A CT image has high-frequency signals. Hence, after the input images were converted into JPG images, a Gauss lowpass filter was used for signal filtering. Each CT image was segmented into rows from top to bottom. This study elaborated on the 100th row for classification analysis, because the signal waveform corresponding to various parts can be shown form CT images in Table 1. Table 1 illustrates the signal waveforms corresponding to various parts. The BPNN has 30 inputs, one layer, and 18 nodes per layer. Consisting of many highly connected nodes, a BPNN is a computing system divided into the input layer, the hidden layer, and the output layer. The operation of a BPNN requires repeated learning by training until each input can be matched with the desired output. Hence, prior to the learning of the neural network, the training pattern should be established to allow the BPNN to implement a learning reference. The performance of a BPNN is directly correlated to the training pattern. Table 1. Comparison of CT image signals for various parts. No. Part Original image Waveform 30 Jaw If the rows close to the 100th row, such as the 101st row, the 99th row, or the 98th row, were used for training and testing, the five major parts, namely the chin, mouth, nose, eyes, and brain, were classified classification results compared with the 100th row are the same. However, if an upper row of the CT image was selected, the upper row reflected the air above the nasal tip and the calculated signal did not fluctuate the five parts will lead to misjudgment. If an excessively low row was used, since the bottom part of each CT image mostly shows the brain, the row only displayed the signal of the brain, with the chin, mouth, nose, and eyes not being classified. If the resolution or patient size changes and the corresponding 100th row signal is not significant, need to reselect position of the row in the CT images. Signals of the 100th row of various parts were converted into 30 characteristic s. The characteristic s were determined as the averages of 10 units on the x axis between 107 and 406. The 30 averages were used as the inputs for the BPNN to represent the changes in signals. This study classified the 385 images of the head into five categories, namely jaw, mouth, nasal part, eye, and brain. By using the 100th row of each image as the training data, this study conducted the BPNN training using 385 images. The training accuracy rate was up to 99.7%, as shown in Table 2. The classifier after learning and training used the 101st row of the images as the testing data for testing the 385 images. The accuracy rate was up to 99.0%, as shown in Table 3. This study used complex and nonlinear signals as the characteristic values. For this reason, the BPNN was used as the classifier. Table 2. Training sample information. 110 Mouth 160 Nasal part 230 Eye 300 Brain type Jaw Mouth Nasal part Eye Brain Sum training result Jaw 79 0 0 0 0 79 Mouth 1 60 0 0 0 61 Nasal part 0 0 65 0 0 65 Eye 0 0 0 75 0 75 Brain 0 0 0 0 105 105 Sum 80 60 65 75 105 385 Recognition rate 98.8% 100% 100% 100% 100% 99.7% Table 3. Testing sample information. type testing result Jaw Mouth Nasal part Eye Brain Sum Jaw 77 0 0 0 0 77 Mouth 3 59 0 0 0 62 Nasal part 0 1 65 0 0 66 Eye 0 0 0 75 0 75 Brain 0 0 0 0 105 105 Sum 80 60 65 75 105 385 Recognition rate 96.3% 98.3% 100% 100% 100% 99.0% 2.2 Nasal vestibule region selection Nasal vestibule region selection was the automatic search processing of nasal vestibule. The non-nasal images classified by the classifier were converted into white images for storage. If the images were judged as being of the nasal part, the image

577 J. Med. Biol. Eng., Vol. 34. No. 6 2014 processing continued to find the information of the representative s of nasal vestibule for grayscale processing, filtering processing, segmentation, masking, image negative, morphological and connectivity marking, and area feature calculation to determine and store the information locations. The experiment flow chart was shown in Fig. 2. Classified images Image enhancement Original image Grayscale processing Image filtering Image segmentation processing to convert the nasal part information for presentation in black and the nasal vestibule location for presentation in white. Since connectivity mark conducted the labeling action of the nasal region, image negative processing was implemented before computation, as shown in Fig. 3(g). Next, morphological calculation of the image was conducted to eliminate the noise for erosion and expansion processing to obtain the image after processing, as shown in Fig. 3(h). The processing results of connectivity marking were shown in Fig. 3(i). The areas of the marked image were compared to find the third and fourth largest areas representing the nasal vestibule information, and centroid calculation of the areas was conducted in order to obtain the nasal vestibule information s. Image masking Image negative Morphology (a) (b) (c) Connectivity mark Third and fourth largest areas NO Eliminate information (d) (e) (f) YES Retain the third largest area Centroid calculation Retain the fourth largets area Figure 2. Automatic nasal vestibule selection flow chart. This study conducted nasal vestibule region selection using the No.168 image, which was classified as an image of the nasal part, as shown in Fig. 3(a). First, grayscale processing was conducted to convert the 3D image data into onedimensional data to obtain the results after processing, as shown in Fig. 3(b). Since the edges of the CT images were the high frequency contour, smooth filtering processing was conducted, as shown in Fig. 3(c). Next, image segmentation processing was conducted, as shown in Fig. 3(d), followed by the image masking processing to keep the interesting blocks. Figure 3(e) showed the masks used in this study. Figure 3(f) showed the results after the AND operation before the implementation of the image negative of images after masking (g) (h) (i) Figure 3. Image processing (1) results. (a) No. 168 image. Image after (b) grayscale processing, (c) filtering processing, (d) segmentation, and (e) masking. (f) Region of interest selection. Results of (g) negative effect, (h) morphological calculation, and (i) connectivity mark search. After the processing of the nasal vestibule data s, MATLAB was employed to design an automatic nasal vestibule select system to reduce the single image selection processing time. The sequential automatic processing of all the CT images can reduce the required time compared to that of manual selection and thus facilitate nasal surgery planning. 2.3 Nasal septum region selection Nasal septum region selection was the automatic search processing of nasal septum. Images classified by the classifier as non-eye images were converted into white images for storage. If the image was determined as being an eye image, image processing continued to find the information of the nasal septum representative s for grayscale processing, filtering processing, edge detection, segmentation, masking, image negative, morphological and connectivity marking, area feature

Three-dimensional Reconstruction of Nasal Regions 578 calculation, and storage. The experimental flow chart was shown in Fig. 4. Fig. 5(k). The location of the centroid was shown in Fig. 5(l). (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 4. Automatic nasal septum selection flow chart. The experiment used No. 210 eye image for processing, as shown in Fig. 5(a). After the image grayscale processing, smooth filtering processing was used, as shown in Fig. 5(b). Regarding the capture of the image details, Canny [13] was used for lateral processing to observe the changes in the edges of the image to present the bones organization. The processing results were shown in Fig. 5(c). Next, the morphological expansion processing of the image was conducted to highlight the edge lines, as shown in Fig. 5(d). Masking was then used, with the mask shown in Fig. 5(e). The AND operation of the mask and image was carried out to keep the nasal septum region, as shown in Fig. 5(f). By implementing the image negative of the masked image, the borders were presented in black. The processing results are shown in Fig. 5(g). Next, the connectivity marking was implemented to distinguish the blocks in the white area to obtain nasal septum information, as shown in Fig. 5(h). The blocks were computed after connectivity marking in terms of area features. The largest and the second largest areas (background) were presented in black, while the details were presented in white. The processed image was shown in Fig. 5(i). The nasal septum and nasal fine bone information were kept. Next, the maximum white pixel value of each row was set as the location of the nasal septum and the nasal septum information was represented in black, as shown in Fig. 5(j). Finally, the morphological calculation of the row was used to enhance the data information, as shown in (j) (k) (l) Figure 5. Image processing (2) results. (a) Input image. Image after (b) smooth filtering processing, (c) lateral processing, and (d) expansion processing. (e) Image mask. (f) Mask calculation. Results of (g) negative effect, (h) connectivity marking, and (i) area calculation. (j) Location of maximum row. (k) Morphological processing. (l) Centroid calculation. The automatic selection of CT images with the nasal vestibule and nasal septum was confirmed by otolaryngology physicians from the Tri-Service General Hospital. The overall recognition rate was 99.7%. 3. Results and discussion The developed software could automatically search for the nasal vestibule and nasal septum information. This study used the BPNN to classify the organization by the different CT waveform. The training and classification results were shown in the Tables 2 and 3, respectively. The above image processing algorithms were used to automatically search for the nasal vestibule and nasal septum information, the selection results were shown in Figures 3(i) and 5(i), respectively. The Avizo software was used to reconstruct the 3D modeling for the nasal vestibule and nasal septum region selection regions. The nasal vestibule, nasal septum and skull data s were combined to provide complete information presented the physician diagnosed. Figures 6(b)-(d) showed front-, right-, and left-view diagrams of the nasal part, respectively.

579 J. Med. Biol. Eng., Vol. 34 No. 6 2014 edge of the medial edge of the orbital cavity was calculated as shown in Fig. 9. The calculated distances were shown in Table 5. Finally, the distance between the nasal septum and the anterior cranial fossa of the brain was measured as shown in Fig. 10. The measured distances were shown in Table 6. (a) (b) (c) (d) (e) (f) Figure 7. Marked s and surgical path image. (g) Figure 6. 3D reconstruction. (a) 3D image of skull. (b) Front, (c) left, and (d) right views of nasal part. (e) Image after reconstruction. (f) 3D image of skull and nasal part. (g) 3D image of nasal part surgical marking (view (a)). (h) 3D image of the nasal part surgical marking (view (b)). (h) 24 15 14 1 This study reconstructed the skull image, and selected the bone threshold values according to the above experiments. As the bones around the eyes were thin, the selection region of eyes should be checked. The eyes bone had not been selection, this study conducted the manual processing of selection. The 3D skull reconstruction results, as shown in Fig. 6(a), could be used to complete the skull 3D image reconstruction. After 3D reconstruction, manual operation was needed to precisely locate the nasal vestibule and nasal septum. The 3D reconstructions of the nasopharynx, nasal cavity, and paranasal sinuses were shown in Figs. 6(b)-(d). Figure 6(e) depicted images after data reconstruction of the skull, nasal cavity, nasopharynx, and paranasal sinuses. Figure 6(f) showed the perspective view of the skull. Figures 6(g) and 6(h) showed the nasal cavity, nasopharynx, paranasal sinuses, and marked s after the removal of facial bones. The representative s of three main risk regions were inferior and medial edge of orbital rim and anterior cranial fossa of brain was shown in Fig. 7. In the 3D reconstruction, the nasal vestibule was green and the nasal septum was blue. The distance between the nasal vestibule and the nasal septum was calculated and marked. The manual marks were shown in red. The relation between the nasal septum and inferior edge of orbital rim were shown in Fig. 8. The measurement results were shown in Table 4. Next, the distance between the nasal septum and the medial Figure 8. Nasal septum data and marked measurement image. Table 4. distance between the nasal vestibule and inferior edge of orbital rim. 1 35.53 13 31.89 2 37.58 14 31.89 3 36.53 15 34.48 4 36.19 16 33.98 5 36.34 17 34.20 6 36.64 18 32.53 7 34.64 19 32.09 8 34.79 20 31.17 9 33.34 21 30.84 10 32.05 22 31.83 11 29.68 23 31.69 12 31.31 24 31.74 5. Conclusion This study proposed a method for the automatic capture of the nasal vestibule and nasal septum in CT images. By digital preprocessing of CT images, the image signal waveforms of the five major parts were found, and the signal changes of the jaw, mouth, nasal part, eye, and brain were analyzed. By using these signal changes as the characteristic values for BPNN training, the BPNN was used as the classifier. The training sample recognition rate of the classifier for head CT images was 99.7%,

Three-dimensional Reconstruction of Nasal Regions 580 and the learning sample recognition rate was 99.0%. The automatic selection of the nasal vestibule and nasal septum requires less time compared to that for manual segmentation. Figure 9. Measurement of nasal septum and marked s. Table 5. distance between nasal septum and medial edge of orbital. 1 33.23 9 39.47 2 36.87 10 41.98 3 41.78 11 42.54 4 43.46 12 40.66 5 46.36 13 45.10 6 24.62 14 22.72 7 18.41 15 15.04 8 32.88 15 1 Figure 10. Image of nasal septum and marked s measurement. Table 6. distance between nasal septum and marked s for brain. 1 24.10 6 26.13 2 24.45 7 28.67 3 25.72 8 24.10 4 26.33 9 16.06 5 26.98 10 19.14 10 1 This study also demonstrated the 3D image reconstruction of the nasal cavity, nasopharynx, and paranasal sinuses, and provided the representative s of potential risky locations, namely the inferior and medial edge of the orbital rim and the anterior cranial fossa of the brain, for nasal endoscopic surgery. The distances between these risky locations and the nasal vestibule and septum were calculated to facilitate preoperative evaluation and enhance safety in surgery. References [1] T. Delmarcelle, Visualizing second-order tensor fields with hyperstreamlines, IEEE Comput. Graph., 13: 25-33,1993. [2] Z. Wang, F. Zeng, H. Li, Z. Ye, Y. Bai, W. Xia and B. Liang, Three-dimensional reconstruction on PC-Windows platform for evaluation of living donor nephrectomy Original Research Article, Comput. Meth. Programs Biomed., 86: 39-44, 2007. [3] R. Biedert, A. Sigg, I. Gal and H. Gerber, 3D representation of the surface topography of normal and dysplastic trochlea using MRI, Knee., 18: 340-346, 2011. [4] R. Wang, D. Yang and S. Li, Three-dimensional virtual model and animation of penile lengthening surgery, Journal of Plastic, J. Plast. Reconstr. Aesthet. Surg., 65: e281-e285, 2012. [5] A. Pham, A. Rafii, M. Metzger, A. Jamali and E. Strong, Computer modeling and intraoperative navigation in maxillofacial surgery, Otolaryngol. Head Neck Surg., 137: 624-631, 2007. [6] M. Zubair, M. Z. Abdullah, R. Ismail, I. L. Shuaib, S. A. Hamid, and K. A. Ahmad, Review: a critical overview of limitations of CFD modeling in Nasal Airflow, J. Med. Biol. Eng., 32: 77-84, 2012. [7] A. Haralambidis, A. Ari-Demirkaya, A. Acar, N. Küçükkeleş, M. Ateş and S. Özkaya, Morphologic changes of the nasal cavity induced by rapid maxillary expansion: A study on 3-dimensional computed tomography models, Am. J. Orthod. Dentofac. Orthop., 136: 815-821, 2009. [8] S. Görgülü, S. Gokce, M. Huseyin and D. Sagdic, Nasal cavity volume changes after rapid maxillary expansion in adolescents evaluated with 3-dimensional simulation and modeling programs, Am. J. Orthod. Dentofac. Orthop., 140: 633-640, 2011. [9] N. Matteo, S. Domenico, M. Luigi, E. Joaquim, A. Isam, S. Guadalupe, B. Joan, F. Enrique and P. Alberto, The use of a three-dimensional novel computer-based model for analysis of the endonasal endoscopic approach to the midline skull base, World Neurosurg., 75: 106-113, 2011. [10] J. Zhu, H. Lee, K. Lim, S. Lee and D. Wang, Evaluation and comparison of nasal airway flow patterns among three subjects from Caucasian, Chinese and Indian ethnic groups using fluid dynamics simulation Respir. Physiol. Neuro., 175: 62-69, 2011. [11] W. Lorense and H. Cline, A high resolution 3D surface construction algorithm, Computer Graphics., 21: 163-169, 1987. [12] P. Guyomarc h, F. Santos, B. Dutailly, P. Desbarats, C. Bou and H. Coqueugniot, Three-dimensional computer-assisted craniometrics: A comparison of the uncertainty in measurement induced by surface reconstruction performed by two computer programs, Forensic Sci. Int., 219: 221-227, 2012. [13] J. Canny, A computational approach to edge detection, IEEE Transactions Pattern Analysis and Machine Intelligence, 8: 679-698, 1986.