A Liver Surgery Simulator Using Full HD Autostereoscopic Displays

Transcription

1 ITE Trans. on MTA Vol. 6, No. 1, pp (2018) Copyright 2018 by ITE Transactions on Media Technology and Applications (MTA) A Liver Surgery Simulator Using Full HD Autostereoscopic Displays Hideki Kakeya 1 (member), Atsushi Yoshida 2, Bin Yang 2, Yukio Oshiro 3 and Nobuhiro Ohkohchi 3 Abstract We present a liver surgery simulator using full-hd autostereoscopic displays. We have developed two kinds of autostereoscopic displays to keep on showing a full-hd 3D image to a viewer who moves freely in front of the display. One is a 3D display based on time-division multiplexing directional backlight and the other is a 3D display based on time-division multiplexing parallax barrier. We have applied the developed simulator using the 3D displays with different specifications to the education of medical students. The result of the questionnaires suggests that 3D visualization is effective and that reduction of crosstalk plays an important role to promote medical use of 3D displays. Keywords: high resolution, medicine, crosstalk, directional backlight, parallax barrier, time-division multiplexing. 1. Introduction We have been engaged in developing high definition autostereoscopic display systems for medical use. For the purpose of medicine, the resolution matters most. Autostereoscopy, which allows the viewers to see 3D images without wearing glasses, is also preferable because it allows eye-contact among the medical staff. Recently, several autostereoscopic displays that attain full resolution of the display panel have been proposed. One way is to use a directional backlight composed of a light guide film and two light sources 1)-3). The drawback of this method is the fixed viewing zone. To enlarge the viewing zone, directionality of backlight has to be controlled to follow the motion of the viewer, which often requires thick optical systems 4)-5). To present a full-hd 3D image to a viewer who moves freely, we have developed two kinds of autostereoscopic displays. One is a 3D display based on time-division multiplexing directional backlight composed of a convex lens array 6)-8) and the other is a 3D display based on Received May 31, 2017; Revised August 30, 2017; Accepted September 21, Faculty of Engineering, Information and Systems, University of Tsukuba (Tsukuba, Japan) 2 Graduate School of Systems and Information Engineering, University of Tsukuba (Tsukuba, Japan) 3 Faculty of Medicine, University of Tsukuba (Tsukuba, Japan) time-division multiplexing parallax barrier 10)-13). So far we have made several prototype systems based on these two methods. The early models had drawbacks such as low intensity and strong crosstalk. In 2015 we started using the prototype 3D display based on time-division multiplexing directional backlight for the education of medical students. Liversim, which is a software to simulate liver surgeries 14), is implemented on a 2D display and the 3D display so that they can be compared with each other. In this paper we introduce the 3D liver surgery simulators we have developed and report the results of the evaluation by the medical students. This paper is organized as follows. In Section 2 the autostereoscopic display based on time-division multiplexing directional backlight is explained. In Section 3 the autostereoscopic display based on time-division multiplexing parallax barrier is explained. In Section 4 the results of the evaluation on the 3D liver surgery simulators based on time-division multiplexing directional backlight are reported. In Section 5 the results of the evaluation comparing two kinds of autostereoscopic displays are reported and the paper is concluded in Section D display with directional backlight The principle of the conventional directional backlight using a convex lens array is shown in Fig. 1. When dot matrix light sources are placed behind the convex lens array so that the distance between them may be equal to 11

2 ITE Trans. on MTA Vol. 6, No. 1 (2018) Fig. 2 Arrangement of the elemental lenses with multiple phase shifts. Fig. 1 Autostereoscopy with a directional backlight using a Fig. 3 Display panels where the even and the odd frame images convex lens array. are mixed. attained. the focal distance of the elemental lens of lens array, collimated directional light is realized. By changing the The first prototype based on this idea was made by using position of the luminous light sources, the directionality two LCD panels that run in the frame rate of 120 Hz, one of light is controlled. When the directional backlight to for the imaging and the other for the dot matrix light each eye alternates synchronously with the alternation source for simplicity. Ideally the dot matrix light source of left-eye and right-eye images on the LCD panel, the should be LED arrays for the reason of energy efficiency. viewer can see a stereoscopic image without wearing Indeed the first prototype had the problem of low intensity special glasses. By increasing the number of light and strong crosstalk, which amounts to 45 %. To overcome sources, multiple viewers can observe the stereoscopic the problem of luminance we made a revised system where image simultaneously 15)-18). the backlight is replaced from 63 W LED arrays to 526 W LED arrays. The conventional system shown in Fig. 1 has two major problems. One is the aberration of the lenses, To solve the problems of crosstalk, we investigated the which causes crosstalk due to the overlap of the light cause of crosstalk to find out the incomplete time- sources for the left eye and the right eye. The other is division multiplexing and the stray light caused by the poor image quality due to the non-uniform luminance lens array composed of Fresnel lenses are the main and the distinct seam of the coarse lens array. Light causes. rays going through a lens generally weakens as the As shown in Fig 3, the LCD panels we used in the first angles become wider, which causes emergence of bright prototype scan the image serially, which causes severe area and dark area in each elemental lens. crosstalk, for the vertical diffuser mixes the backlight of The aberration can be reduced by inserting a large the odd frame and the even frame. (Here a read image convex lens behind the LCD panel 7). To remove the and a green image are shown alternately on the screen.) seam of lens and the non-uniformity of intensity from When the LCD panels switch the image in a parallel the image, we combine a lens array with various phase manner by dividing the panel areas, the image of the shift and a vertical diffuser 8). The elemental lenses are odd frame and the even frame do not appear at the same aligned so that the phase of lens placement in each row time as shown in Fig. 4. In the second prototype, we may differ from one another as shown in Fig. 2. Since used these LCD panels and measured the crosstalk with the seam of the lenses appear only once in the vertical a luminance meter to find out the crosstalk level direction, the vertical diffuser can smooth the seam dropped from 45 % to 18 %. The mechanism of stray light caused by the grooves of enough and the image with homogeneous intensity is 12

3 Invited Paper» A Liver Surgery Simulator Using Full HD Autostereoscopic Displays Fig. 4 Display panels where the even and the odd frame images are separated. Fig. 5 Stray light that emerges in the Fresnel lenses. Fresnel lenses is explained in Fig. 5. One way to remove the stray light is to narrow the viewing angle of the backlight panel so that the light from a steep angle may be cut. To achieve this we placed a privacy filter on the surface of the LCD panel that works as the dot matrix light source and measured the crosstalk to find out that the crosstalk level dropped from 18 % to 13 %, while the luminance is also reduced due to the addition of the filter. Fig. 6 Time-division quadruplexing parallax barrier. 3. Time-division multiplexing parallax barrier alternately at a refresh rate of 120 Hz 19)20). These systems, however, cause crosstalk when the viewer is In the conventional parallax barrier system, the not located at the proper viewing point. viewing position is fixed and the resolution is half that of the original display panel. Time-division duplexing To achieve high resolution and expansion of the parallax barrier solves the problem of resolution loss viewing areas free from crosstalk, time-division caused by the conventional parallax barrier. The time- quadruplexing parallax barrier has been proposed (Fig. division duplexing parallax barrier systems show two 6) 10). Here 4 viewpoints and the corresponding images kinds of images and parallax barrier patterns are denoted as A, B, C and D. The slit is shifted by one 13

4 ITE Trans. on MTA Vol. 6, No. 1 (2018) pixel at each frame and drawing of a full resolution image is completed after 4 frames. Though flicker emerges because of the quadruple time-division, it does not stand out when the slits of the parallax barrier are as fine as 1 pixel wide. The flicker can be further suppressed by applying anaglyph parallax barrier to this system 9)13). The viewing zone without crosstalk can be extended with "L-L-R-R" alignment in the 4 view system, where the left-eye image is delivered to 2 viewpoints and the right-eye image is delivered to the other 2 viewpoints. The left-eye image is shown at points A and B, and the right-eye image is shown at points C and D. If the left eye is between points A and B and the right eye is between points C and D, 3D images without crosstalk can be observed. By changing the slit position in accordance with the viewer's position by subpixel unit, continuous stereoscopy is maintained 11)12). The prototype based on this method reduce the crosstalk level to 6 %. 4. Evaluation of 3D surgery simulator based on time-multiplexing directional backlight Liversim is a real-time virtual hepatectomy simulation software that provides 4 basic functions: viewing 3D models from arbitrary directions, changing the colors and opacities of the models, deforming the models based on user interaction, and incising the liver parenchyma and intrahepatic vessels based on user operations (Fig. 7). We implemented Liversim on the 3D displays based on time-division multiplexing directional backlight we introduced in the previous section as shown in Fig. 8. To evaluate the effectiveness of the 3D displays, we asked medical school students to answer the following questions after trying both 2D version and 3D version of Liversim: (Q1) Do you feel depth in the 3D version? (Q2) Which gives better understanding of liver deformation? (Q3) Which gives better understanding of blood vessel stream? (Q4) Do you think autostereoscopy is helpful? First we asked these questions to 24 medical students from age 21 to 30 (13 male, 11 female) who tried 2D Liversim and 3D Liversim based on the first prototype (45 % crosstalk, 63 W backlight) during the educational program from August 17 to October 13, In the first prototype, a pair of 24 inch TFT panels from BenQ XL2420 were used for imaging and the dot matrix light control. The lens array was composed of 650 elemental Fig. 7 A screen shot of Liversim. Fig. 8 Liversim implemented on the 3D display based on timemultiplexing directional backlight. lenses (20 mm wide and 12 mm high) whose focal distance was 30 mm. A large convex Fresnel lens whose focal distance was 800 mm was placed behind the imaging panel. Kinect v2 was used for head tracking. Next we asked the same questions to 28 medical students from age 21 to 30 (18 male, 10 female) who tried 2D Liversim and 3D Liversim based on the second prototype (18 % crosstalk, 526 W backlight) during the educational program from February 23 to May 9, In the second prototype, a pair of 24 inch TFT panels from Asus VG248 were used for imaging and the dot matrix light control. The same lenses and the headtracking device were used in the second experiment also. Figs. 9 through 12 show the results of the evaluation by the medical students. As these figures show, the second prototype attains better scores in every question. Chi-square tests show that improvement is statistically significant where p values are all far below 0.01 in every question. Also almost all the students answer that 3D is better than 2D and no students answer that 2D is better than 3D in questions 2 and 3 after the intensity of backlight is increased and the crosstalk is decreased as shown in 14

5 Invited Paper» A Liver Surgery Simulator Using Full HD Autostereoscopic Displays Fig. 9 Responses to Q1 in the 1st prototype (left) and the 2nd Fig. 13 Liversim implemented on time-division multiplexing parallax barrier. Fig. 10 Responses to Q2 in the 1st prototype (left) and the 2nd Fig. 14 Medical students trying systems A and B. Fig. 11 Responses to Q3 in the 1st prototype (left) and the 2nd multiplexing parallax barrier, we implemented Liversim on the prototype display system as shown in Fig D display based on time-multiplexing parallax barrier is labeled as A, while the 3D display based on timemultiplexing directional backlight is labeled as B, both of which are placed side by side for the use of medical students as shown in Fig. 14. We asked 16 medical students from age 21 to 24 (9 male, 7 female) to compare Fig. 12 Responses to Q4 in the 1st prototype (left) and the 2nd them on Dec. 6, 2016, Jan. 10, 2017, and May 16, On the first two days the resolution of the image given by time-multiplexing parallax barrier was half HD for Figs. 10 and 11. Thus the effectiveness of 3D displays for the software reason, while the resolution was full HD on the surgery simulator is confirmed. the last day. We asked the following questions: (Q1) Which gives a better image quality? (Q2) Do you 5. Evaluation of 3D surgery simulator based on time-multiplexing parallax barrier perceive double image? (Q3) Which display do you want to use? Figs. 15 through 17 show the result of the As the result in the previous section shows, reduction questionnaire. As shown in Figs. 15 and 16, no of crosstalk improves the evaluation by the medical significant difference between A and B was found in students. Therefore use of time-multiplexing parallax questions 1 and 2. The crosstalk level measured barrier is expected to obtain better response by the objectively is far lower in system A than in system B. students, for the crosstalk is further suppressed. The reason why significant difference was not found in question 2 may come from the head-tracking problem, To evaluate the 3D surgery simulator based on time15

6 ITE Trans. on MTA Vol. 6, No. 1 (2018) Fig. 18 A professor exlaining the hepatic sements to a medical student. Fig. 15 Responses to Q1 comparing systems A and B. 6. Conclusion Fig. 16 Responses to Q2 on system A (left) and system B (right). In this paper we have presented a liver surgery simulator using full-hd autostereoscopic displays. We have developed and have used two kinds of autostereoscopic displays that keep on showing a full- HD 3D image to a viewer who moves freely in front of the display. We implement Liversim, a liver surgery simulator, on the 3D displays with different specifications for the purpose of medical education. The result of the questionnaires to medical students suggests that 3D visualization is effective and that reduction of crosstalk plays an important role to promote medical use of 3D displays. Acknowledgement This research is supported by the Grant-in-Aid for Scientific Research, JSPS, Japan, Grant number: 17H00750, and by Casio Science Promotion Foundation, Grant number: References Fig. 17 Responses to Q3 comparing systems A and B. for the camera for head-tracking was placed at the bottom of the display for hardware reason, which more often fails to track the face of the viewer. Fig. 17 shows that system A is evaluated higher than system B for practical use, which may reflect the evaluation of lower crosstalk. Fig. 18 shows the actual educational scene where a professor is explaining how the 8 hepatic segments can be distinguished by the branching of blood vessels. As this figure shows, eye-contact between a professor and a medical student is enabled due to the autostereoscopy, which helps smooth communication between them. 1) J.C. Schultz, R. Brott, M. Sykora, W. Bryan, T. Fukamib, K. Nakao and A. Takimoto "Full Resolution Autostereoscopic 3D Display for Mobile Applications," SID 2009 Digest, pp (2009) 2) A. Travis, N. Emerton, T. Large, S. Bathiche and B. Rihn, "Backlight for ViewSequential Autostereo 3D," SID 2010 Digest, pp (2010) 3) M.J. Sykora, "Optical characterization of autostereoscopic 3D displays," in Stereoscopic Displays and Applications XXII, Proc. SPIE. vol. 7863, 78630V(2011) 4) T. Hattori, et al., "Advanced autostereoscopic display for G-7 pilot project," SPIE Proc. 3639, pp.66-75(1999) 5) A. Hayashi, et al. "A 23-in. full-panel-resolution autostereoscopic LCD with a novel directional backlight system," Journal of the Society for Information Display, 18, pp (2010) 6) S. Ishizuka, T. Mukai and H. Kakeya, "Viewing zone of an autostereoscopic display with a directional backlight using a convex lens array," Journal of Electronic Imaging, 23, 1, pp (2014) 7) T. Mukai and H. Kakeya, "Enhancement of viewing angle with homogenized brightness for autostereoscopic display with lensbased directional backlight," in Stereoscopic Displays and Applications XXVI, Proc. SPIE vol. 9391, 93911A(2015) 16

7 Invited Paper» A Liver Surgery Simulator Using Full HD Autostereoscopic Displays 8) S. Ishizuka, T. Mukai and H. Kakeya, "Multi-Phase Convex Lens Array for Directional Backlights to Improve Luminance Distribution of Autostereoscopic Display," IEICE Trans. Electron., vol. E98-C, No.11, pp (2015) 9) Q. Zhang and H. Kakeya, "An Autostereoscopic Display System with Four Viewpoints in Full Resolution Using Active Anaglyph Parallax Barrier," in Stereoscopic Displays and Applications XXIV, Proc. SPIE. vol. 8648, 86481R(2013) 10) Q. Zhang and H. Kakeya, "A High Quality Autostereoscopy System Based on Time-Division Quadplexing Parallax Barrier," IEICE Trans. Electron., vol. E97-C, No.11, pp (2014) 11) K. Okada and H. Kakeya, "Autostereoscopic Display System Using Quadruple Time-Division Multiplexing Parallax Barrier with Subpixel Structure," Proc. IDMC '15 3DSA, AP3-001(2015) 12) H. Kakeya, H. Takahashi and K. Okada, "Parallax based autostereoscopic display with a deep viewing zone," Proc. IDW, 3DSA3/3D3-2(2016) 13) Q. Zhang and H. Kakeya, "Time-division quadruplexing parallax barrier employing RGB slits," Journal of Display Technology, 12, 6, pp (2016) 14) Y. Oshiro, et al. "Novel 3-dimensional virtual hepatectomy simulation combined with real-time deformation," World J Gastroenterol. 21, 34, pp (2015) 15) T. Hattori, "Stereoscopic Picture Display Device," JP Patent , A(1996) 16) T. Hattori, "Stereoscopic Picture Display Device," JP Patent , A(1996) 17) T. Hattori, "Stereoscopic Video Display Device," JP Patent , A(1996) 18) T. Hattori, "Stereoscopic Video Display Device," JP Patent , A(1996) 19) K. Perlin, et al., "An autostereoscopic display," Computer Graphics, pp , 2000, 20) H.J. Lee, et al., "A high resolution autostereoscopic display employing a time division parallax barrier," SID 06 Digest, pp.81-84(2006) Hideki Kakeya He received a doctoral degree in engineering from the Univ. of Tokyo in He now is an associate professor of the University of Tsukuba Atsushi Yoshida He received a master degree in engineering from the University of Tsukuba in Bin Yang He received a master degree in engineering from the University of Tsukuba in Yukio Oshiro He received a Ph. D. degree in medicine from the University of Tsukuba in He now is an assistant professor of the University of Tsukuba. Nobuhiro Ohkohchi He received a Ph. D. degree in medicine from Tohoku University in He now is a professor of the University of Tsukuba. 17