Texture Generation for the Computer Representation of the Upper Gastrointestinal System

Transcription

1 A. Gastélum, P. Delmas, J. Márquez, Texture Generation for the Computer Representation of the Upper Gastrointestinal System, Proceedings of Image and Vision Computing New Zealand 2007, pp , Hamilton, New Zealand, December Texture Generation for the Computer Representation of the Upper Gastrointestinal System Gastélum Alfonso 1, Patrice Delmas 1 and Márquez Jorge 2 1 University of Auckland, Tamaki Campus, New Zealand 2 Laboratory of Analysis of Images and Visualization, CCADET, UNAM, México valdenar@gmail.com, agas012@ec.auckland.ac.nz Abstract We previously introduced a system to obtain textures using the lens system of a video-endoscope; in this work we present our most recent advances. Our goal is to build a system which obtains textures for our 3D model of the oesophagus and in a later stage, to build new meshes from individual video endoscopies. The importance of obtaining a database of textures for our model is related to the necessity of training new specialist in endoscopies for the detection of diseases characterised by abnormal colour patterns. The colour extraction is based on the physical properties of the endoscope s lens and associated illumination system. We obtain a group of 3-dimensional coordinates with RGB values that will be related to the actual 3D model. To relate texture as captured from endoscopy videos with actual depth (z coordinate) we used a second camera to record the external view of the endoscope insertion. Keywords: textures, mesh colour diseases representation, mesh building. 1 Introduction Computer training in endoscopical procedures allows the specialist to interact with a virtual model and provides different points of view of the anatomical area of interest. Such enriched navigation permits the specialist to have a better understanding of the whole anatomical volume. To complete a computational training system with our upper gastrointestinal model and navigator, reported in [1], we developed a navigation environment that allows a user to explore the model and train on anomaly detection. Computer models for endoscopies must provide a realistic environment [4], one important part being the texture. The endoscopic procedures are strongly related to optical inspection. The availability of a library for common disease textures will help the training of new specialists. The goal of this work is to present a method which obtains the above mentioned textures and corresponding depth information and map them to our actual 3D model. The navigation system may challenge the specialist with various near real case scenarios. 2 Problem The original texture of our model was obtained from the Visible Human Project (VHP) data base [2]. As the colour of the images in the data base was altered by the post-mortem condition, it does not offer a realistic texture representation of the oesophagus (as shown in Fig. 1). Figure 1: a) Real endoscopy image, b) Model of the oesophagus mapped with the colours extracted from the VHP database. To overcome this, we replaced the original texture by more realistic ones computed from video-endoscopies of the oesophagus. 305

2 The procedure comprises four stages: The arrangement of the cam recorders is shown in figure The preliminary depth values are obtained using the properties of the lens system. The external webcam is placed above the head of the patient, so it can record all the procedure without been obstructed by the specialist or the patient. This camera provides a reading of the amount of endoscopic tube being inserted in the patient, as shown in Fig In order to obtain a better depth value a procedure relating the luminance of a pixel to the depth has been developed The images are weighted in order to decide if they present useful information 4. - The results are processed to obtain a final RGB value for our textures. a 3 Procedure Since our model is built to serve as a training system, the navigation must provide the closest possible realistic environment. One improvement is to present textures that resemble the real ones. The video camera at the tip of the endoscope records the inner walls of the oesophagus but there is no inbuilt routine way to record the camera s depth inside the patient. Figure 3: Image from the external video taken with a webcam. Point (a) shows the depth marks in the endoscope tube. Redo figure 3 with just an arrow pointing to depth mark and the letter a It was therefore necessary to use an external video camera to capture the depth values as inscribed in the endoscope tube side. The video from the endoscope camera provides the internal view of the organ (Fig.4). The z-coordinate assigned to each frame is the one obtained from the external video camera. After obtaining the endoscopy and endoscope tube depth videos, we built a timetable that related them. The external camera provides the z-coordinate for each frame while the video-endoscopies provided the RGB information. The depth value and the video-endoscopies are used to obtain a collection of concentric isolines. They depend on the local illumination and the lens distortion for each frame. Isolines are used to obtain the final colour map. Figure 4: Image from the internal walls of the oesophagus D coordinates projections In order to obtain a texture from the collection of 2D frames, we use the optical properties of the endoscopes lens and the light propagation phenomena to obtain luminance isolines that represent the depth in each individual frame. The objective is to obtain from each frame a collection of points that gives the (x, y, z) position coordinates and associated RGB value. The z coordinate is constructed from the initial depth (as given by the external video camera) and depthdependant classification of the endoscopic images luminance value. Figure 2: Endoscopic experimental arrangement. (a) Is the external video camera 3.1 Optical System The first step consists in the synchronised acquisition of the video endoscopies and the external support video. This gives us the opportunity to build a timetable that related both videos frame by frame. Figure 5 shows an example (similar to the endoscopic case) of a 3D surface plot obtained with Imagej and the plugin Imagej3D. The surface plot takes the intensity value as a representation of depth. 306

3 Figure 5: Example of a 3D surface build using intensity values of a 2D image. 3.3 Endoscope Lens Optical System The properties of the field of view for wide angle lens ([3], [5]) are used as one of the parameters to map 2D image vertices to 3D point representation. We consider the ideal case of a cylinder where the greater the angle of view, the closer the object gets to the lens, along the Z axis (Fig. 6): z a (1) Where is the angle of view and a is a scalar factor from the lens. 2. Since the centre of the endoscope and the oesophagus are not the same, we lose the symmetry as shown in the cylinder model (Fig. 6b) The presence of tissue folds is the most difficult problem related to segmentation, because we cannot characterise 3D folds using only one image, and occluded regions will appear as black shadows in our model. To map the colour value from the images to the vertex table, the distance from the correct centre to the pixel of interest is calculated. All the pixels having the same distance will belong to a contour. Each one of these contours will have the same depth value. d Figure 7: Contour formed by pixels at the same distance from the centre. Figure 6: a) Ideal representation of the oesophagus. The sphere represents the lens position. b) Side view of the cylinder, shows the relation of the angle of view with the depth This gives us the z-coordinate first approximation for each pixel. Later on (see section 2.2.2) we will use the luminance value of endoscopic images, to improve the z-coordinate value estimation for any given pixel of the oesophagus. There are important differences between a real oesophagus (images show in figure 4) and a cylindrical model. Three problems occur when analyzing real endoscopies images: 1. - The centre of the anatomical structure (oesophagus) and the centre of the lens are not always the same. To correct this, we need to find a new relation for the centre, using a correcting term x. From each representative distance we obtain a list of pixels, the number (N) of pixels that belong to a certain contour will vary. The number of contour (N c ) obtained depends on the spectrum of angle of view values used and the separation between angles value. Each contour will belong to an specific angle of view value: Max( ) Min( ) N c m Where m is the number of quantization steps. (2) As explained in point 1 of our problems, the lens may not be positioned in the centre of the oesophagus or its shape will not be cylindrical and symmetrical. To account for these cases we have to obtain a new centre for the image which will help us reinstate the oesophagus cylindrical symmetry paradigm. Taking advantage of the cylindrical symmetry of the endoscopic images intensities we traced plot profiles from the centre of the image, at different angles as shown in figure 8. From the plot (see Fig.8 d) we obtain the distribution of the zone with the lowest intensity values and the centre of this zone becomes the centre of the image. 307

4 We first transform from the red, green, blue (RGB) colour space into the HSL colour space: Figure 8: a) Original image b) Gaussian filtered images c) Profile lines d) Plotted profile resulting from different angles 3.4 Lightness colour Space. Next, we study the endoscopic frames colour properties in the Hue, Saturation and Lightness (HSL) colour space. The only sources of illumination, inside the oesophagus, are the two light-guide lens located at the tip of the endoscope (Fig 9). We next use the information of this channel to obtain a better representation of the depth properties in the image. Figure 9: a) Endoscope distal tip. Since the illumination system can be seen as two punctual source of light, we can evaluate the decay of the intensity of light to characterise our 2D images. Considering reflection free images, the intensity value at different distances is described in figure 10. The values were obtained from the manufacturer (Moritex) of the light-guide lens. Figure 11: a) Process for obtaining the L channel from each from of the video-endoscopies. From left to right: a) original RGB image; b) image showing the L channel; c) same image with a Different LUT (ImageJ fire ) From each of the resulting L-channel image we extract iso-contours having the same L-value. To obtain a better result we first applied a Gaussian filter to the images to remove small changes in lightness. Next, a z-value is assigned to each pixel depending on its L-value. Figure 12 shows a schematic view of the use of the L- value. The coordinates forming the surfaces are at identical (x, y) image coordinates while z is given by their L-value. Next, each group of coordinates is transformed to polar coordinate system. Figure 10: a) Cone representation of the endoscope light system. Considering this behaviour we can assume that the intensity of the light in each pixel will predominantly depend only on the distance between the light source and the object at the moment of the recording. Figure 12: Depth field map obtained from the image L channel. From left to right: a) isolines representing the light changes; b) resulting surface; c) surface with the RGB colour as texture. d) Different angle of view. 308

5 3.5 Image Classification. For the system to obtain isolines that classified the depth at which a pixel is in frame, the frame must fulfil some properties. The selection of the frames is important so the process does not consider frames that will not introduce new texture info to the averaged RGB colour. First the algorithm checks if it is possible to obtain a centre for the images. The images must exhibit a darker area which ensures depth. Next, we check if the images have sufficient isolines to differentiate enough depth levels. To do so the algorithm computes the L-channel histogram and its standard deviation. To be kept, an image histogram must present a Gaussian-like distribution. Figure 15: a) top: Original images showing a frame where the lens is to close to the esophagus wall; Down: L-channel; Right: Isolines b) Above: Original Highly asymmetric image; Down: L-channel; Right: Isolines c) Above: Original saturation of light in the corner might resemble symmetry but the frame is discarded; Down: L-channel; Right: Isolines. Figure 13: Endoscopy image and corresponding histogram. From top to bottom: rejected frame; b) accepted frame. Finally after re-centring the images, the isolines must fulfil the following heuristics: 1) The isolines with bigger radius must have also the highest lightness value. 2) If the isoline has a small radius and a high value of lightness then the isoline must belong to the corners of the image 3.6 Texture model Now we have a system to obtain isolines from our images. We also have two different ways to calculate the z-values for each frame of the video-endoscopies. The next step is to build the texture model. First, we applied the procedure described in points and to all the frames in the videoendoscopies. For each frame we obtain a group of pixels with angle and depth values, one from the lens analysis and one from the lightness. We build a data table containing the angle and the total depth value (initial depth of the endoscope distal tip plus the one obtained for each contour of the image). At this point we do an average of the depth from each technique, and after that a final average for all the points that have the same total depth. In the end we have a group of points with angle, depth and RGB values. The 3D triangle mesh model can be segmented in 2D contours; we select the contour of interest using the initial depth given by the endoscope tube values plus each one of the d values from the images. Figure 14: Endoscopy images and corresponding isolines. From top to bottom: a) original image, b) re-centred image. We project the pixel to our 3D model using the angle from each pixel and the radius from the 2D contour. The coordinates of the pixels in the 3D model would be: 309

6 p R q R c c cos( ) sin( ) (4) The vertex positions that are not mapped from the pixels will obtain theirs RGB values from the colour value of the neighbours by bilinear interpolation. Figure 16 shows different stage of the newer texture been mapped on to the model The frame classification described in section 3.5 allowed to better cluster suitable and unsuitable images for texture retrieval purpose. This improved the accurate detection and discarding of images not giving information related to the RGB value, due to light saturation, (losing anatomy) or lack of symmetry. The next step is to build a comprehensive library of different diseases represented by the colour textures obtained from different video-endoscopies and map them to our system. 5 Acknowledgements We would like to thank Dr Jose Luis Mosso for providing the video-endoscopies. Figure 16: Different stages of the colour mapping over the model. After obtaining a unique RGB value for each z- coordinate and angle, we need to eliminate the effect of differences in illumination. We do a transformation from the initial RGB value of each pixel to the HSL value and normalize all L values to a single value. Our 3D navigator simulated the lighting as emitted from the endoscope tip. We combined the original Hue and Saturation pixel value with the L value obtained by the simulated light. References [1] Gastélum Alfonso and Márquez Jorge, "Construction of a model of the upper gastrointestinal system for the simulation of Gastroesophagoendoscopic procedures". VIII Mexican symposium on medical physics, American Institute of Physics 724: [2] V. Spitzer, M. J. Ackerman, A. L. Scherzinger, and D. Whitlock, The Visible Human Male: A Technical Report. J. of the Am. Medical Informatics Assoc 3(2): [3] Tsai RY, A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J Robotics 3: Figure 17: a) Real endoscopy image, b) Model of the esophagus mapped with the colour from a videoendoscopies. 4 Conclusions We described current work to incorporate a disease or healthy-related colour texture to our computer model of the oesophagus. This information is obtained from actual video-endoscopies, with the goal of providing the user with a more realistic experience when using our model. Another advantage is that, in real procedures, it is very important for the specialist to be able to distinguish the colour pattern of some disease like GERD [6]. Building such library will allow the user to train in the detection of different anatomical conditions. [4] Morten Bo-Nelsen, Joseph L. Tasto, Richard Cunningham, Gregory L. Merril, Preop endoscopic simulator: A pc-based immersive training System for bronchoscopy. Studies in health technology and informatics 62: [5] F. Devernay and Faugeras, Straight Lines Have to Be Straight: Automatic Calibration and Removal of Distortion from Scenes of Structured Environments. Machine Vision and Applications 1: [6] ASGE Publication, The role of endoscopy in the management of GERD. Gastrointest. Endosc.49(6):