A Haptic-Assisted Guidance System for Navigating Volumetric Data Sets

Transcription

1 A Haptic-Assisted Guidance System fo Navigating Volumetic Data Sets Eick Méndez*, Shunsuke Yoshida, Hauo Noma, Robet W. Lindeman*, Yasuyuki Yanagida, Shinobu Masaki, Kenichi Hosaka (*)Compute Science Depatment, Geoge Washington Univesity, Washington DC, USA ( )Media Infomation Science Laboatoies ATR, Kyoto, Japan ( )Bain Activity Imaging Cente ATR, Kyoto, Japan {mendex gogo}@gwu.edu, {shun noma yanagida masaki hosaka}@at.jp Abstact In this pape, a new appoach fo navigation assistance though sets of volumetic data is pesented. This innovative method uses foce feedback in two dimensions to guide the use to a egion of inteest. The haptic feedback is calculated unde two methods and we explain the implementation of both appoaches. The output is displayed by the Poactive Desk, a two dimensional foce feedback device developed at the Advanced Telecommunications Reseach Institute. This new appoach is tageted to scientists who wok with sets of volumetic data and who wish to find infomation using visual scanning enhanced with haptics. Ou system is designed fo Magnetic Resonance Images (MRI) ovelaid with functional MRI. 1. Intoduction Visualization of data has always been an impotant subject in the epesentation of infomation. It povides a fast way to localize data that could be impotant fo the use. Fo the past two decades, data has become so complex and vaied that a thee dimensional epesentation is necessay. With the visualization of fluid dynamics, textue, and medical images, the field has boadened within the computational ea. As noted by Lundin et al. [9], when woking on medical data, analysis can benefit fom the use of haptic feedback and the task execution speed is inceased. In the aticle by Haan et al. [5], the authos note that even though people ae used to woking in the thee dimensions of the eal wold, when pefoming tasks like dawing o measuing, they usually seach fo a suppoting plane of efeence. In addition to the epesentation of the data, localization of specific data is a poblem itself. In thee-dimensional epesentations, occlusion is a common poblem. This poblem occus when two o moe objects ovelap in the line of sight of the use, and the closest object occludes those fathe away. This is, theefoe, a common issue in theedimensional visualization of medical data. The use of Magnetic Resonance Imaging (MRI) and Functional MRI (fmri) has inceased ove the past decade. fmri is used to locate egions whee activity of the bain is detected. These egions ae of high inteest to eseaches and ae a common subject of study. This activity is displayed to the use visually by ovelapping the functional scan on top of the base MRI. An example of the poblem of occlusion is shown in Figue 1. As seen in the figue, bain tissue is occluded by the ovelapping of the fmri. It is ou intention to povide new ways of localizing infomation than by only poviding visual cues. Occlusion can be solved by the use of semitanspaent objects as noted by Iwata and Noma [8], howeve, multiple objects may ovelap in the image. Moeove, the aea of inteest may not be lage enough fo the use to easily spot it. In ou cuent wok, a new appoach fo navigation is pesented. Ou eseach is focused on the task of finding egions of inteest ove sets of volumetic data. The solution pesented uses foce feedback to guide the use s hand towads the egion of inteest. The device fo the display of foce feedback that we use is the Poactive Desk (PD) by Noma et al. [10]. The PD is especially suitable fo this poject fo the elatively lage aea that it can cove compaed to othe devices. Also, an impotant fact is that it woks ove a two-dimensional plane. Two methods fo haptic

2 endeing wee developed and we will explain the implementation of these in the following sections. Figue 1: An example of the poblem of occlusion of fmri data ovelaid on an MRI scan. 2. Related Wok Seveal appoaches fo volumetic epesentation by haptics (haptization) have been developed. The fist method fo haptization of volumetic data was intoduced by Iwata and Noma [8]. Subjects wee given the task of finding egions of high density data by poviding them with visual and haptic feedback. Results showed in this task, that haptic feedback inceased the accuacy of the subjects. The ate of eo deceased fom 35% in the visual teatment, to 19% in the visual+haptic teatment. In the study by Iwata and Noma, the use was always pesented with visual infomation. No haptic guidance assistance was povided. Moe ecently Ikits et al., fom the Univesity of Utah, intoduced an algoithm that esticts the motion of a poxy in cetain diections when woking on volumetic data [7]. The goal of this eseach was to help the use to exploe the volume. The haptized data used was heat muscle and bain tissue. Howeve, no wok was done in guiding the use towads specific data, fo example, a bain tumo. Reimesdahl et al. intoduced endeing techniques fo the exploation of Computational Fluid Dynamics data in vitual envionments [13]. In this wok a simila appoach to one of ou haptic endeing methods was used. In thei gavity scalas method, points in the data gid ae assigned a mass. This will attact the pobe object to those egions of high density; howeve, this wok uses a diffeent haptic device, a PHANToM. Linea tajectoies fo paths fom outside the head to a taget inside the bain wee implemented by Hinckley et al. [6]. Hee the use holds a pop that acts as the cutting plane of the data set, howeve, no active haptic feedback is given to the use. Moeove, no guidance assistance is povided. Navigation and manipulation of 2D and 3D data have been tied befoe, and an excellent appoach is that of Aliakseyeu et al. [1]. In this eseach a twodimensional desk is povided to the use, whee he is able to manipulate the data. The goal of the eseach was the exploation of thee dimensional datasets. Infomal esults showed that the system added value in the task pefomance. Howeve, again, no active haptic feedback was povided to the use. At the Univesity of Delft, Haan et al., ceated anothe appoach fo volume exploation employing passive feedback using a pad and a stylus [5]. This wok is a good appoach fo volume slicing. In thei eseach, howeve, the use ties to get an insight of the data by inspecting o pobing the vaious data values. They do not povide any active foce feedback fo navigation. Some techniques deal with the haptization of the infomation, static o dynamic, while othe techniques deal with the estiction of movement of the poxy. Few techniques have woked on the task of finding infomation by navigation assistance. 3. The Haptic Feedback Device The Poactive Desk [10] is a foce feedback device that woks by applying foces to electo-conductive objects ove its suface, the desk (Figue 2). A pojecto, placed above this desk, is used to display images on its suface. The use can now feel haptic foces and, at the same time, see an image that coelates with the foces that he feels. The PD has its egion of effect in the cente of the desk of dimensions equivalent to a 512x512 squae of pixels. Theefoe, we pesent the image of the sliced data in the cente and of exactly the same dimensions. The est of the suface is used to suppot non-haptic inteface widgets. Ou system uses a stylus fo the navigation aid. The stylus we use has an aluminum disk attached to its tip. The colo of this disk matches the colo of the suface of the PD. By doing this, the pojected image blends and the use can peceive only mino distotion. This stylus sits on top of the PD s suface and it will move accoding to the foces applied by the PD. By holding the stylus in his hand, the use can feel sensations such as boundaies, viscosity, o foces which, in ou case, ae used fo guidance. On the uppe ight cone of Figue 2 is a pictue of the stylus with the aluminum disk attached to its tip. The stylus is a wieless pop; theefoe, its position has to be ecognized by an optical senso also positioned above the desk. The stylus position is tansfomed to mouse coodinates which ae used to guide a computed pobe object. Given the position of this pobe we use the appopiate foce that the PD will apply to the stylus.

3 navigation task. This can be thought of as a meging of the ideas of Lundin et al. [9] and Aliakseyeu et al. [1] System Oveview Figue 2: The Poactive Desk. On the uppe ight cone, we also pesent a pictue of the stylus. 4. Guidance System We pesent a solution to the poblem of localizing data by using, as an example, ovelapped fmri data. Infomation of activity in the bain is obtained by giving stimuli while pefoming an MRI scan. We define ou Regions of Inteest (ROI) as those egions whee activity was detected by the fmri. Ou eseach pesents the implementation of a system capable of guiding the use towads the ROI depending on the distance to the egion, o the distance and density of the taget egion. This system uses haptic feedback to push the use towads the desied diection. We calculate vectos that will povide us with the amount of foce and diection necessay to guide the use. It should be noted that this technique should be applicable to othe volume data sets as well, such as those used in the petoleum industy. The foce vectos ae pe-computed and stoed in a Foce Vecto Mesh (FVM). The system ecognizes the two-dimensional position of the stylus and maps it to the thee-dimensional volume. Once knowing this position, the system only pefoms a lookup in the stoed FVM. With the esulting foce vecto, the PD can now guide the use in the necessay diection. A special case occus when the taget is on a paallel plane. We ae cuently dealing with ways of solving this situation. Ou system is capable of slicing the volume in any abitay position and oientation. Fou planes ae povided fo this task: thee othogonal, axis-aligned planes, and one of abitay position and oientation. Any of the fou may be chosen fo navigation. Aliakseyeu et al. incopoate the idea of "natualness" by defining five guidelines that an inteaction device should follow [1]. Ou system complies with these five guidelines and, in addition, we povide haptic-assistance to help the use on the An image of the system we have developed is pesented in the Figue 3. The ROIs ae maked in geen and blue in the cente image. This cente image is whee the foce vectos will be applied. While the use has the stylus in this aea, he will be able to feel the PD guiding him towads the desied diection. As mentioned befoe, thee ae fou planes that ae used to slice the data: thee othogonal, axes aligned planes and a fouth that we call the abitay cutting plane. This abitay plane lies fixed on a vitual floo. When tansfomations ae applied to the data, such as otations o tanslations, they will affect the way the plane intesects the data. The use can move the othogonal planes by physical dials that we povide on the bottom of the PD, o by using widgets on the ighthand side of the wok space. As seen in the figue, we pesent a high-esolution, 512x512 pixel aea in the cente of the wokspace. This is whee the main display of infomation occus, and this is also the aea of effect of the PD. Since we give fou diffeent planes, we also povide a way of selecting one of these by widgets o buttons on the tackball descibed late in this section. On the left-hand side of the PD, a thee-dimensional epesentation of the MRI and fmri data is pesented as a wie fame. The othogonal cutting planes ae shown in gay. The abitay cutting plane is shown as the gay semi-tanspaent floo. Manipulations of the wie fame such as otations o tanslations ae done by a tackball. The use can eoient the data by moving the sphee of the device. By doing this, the abitay plane that acts as a floo will cut the data in diffeent oientations. The system povides widgets fo these tasks that can be accessed by the stylus. The tackball is intended as a goss manipulation method by the non-dominant hand of the use. Fine manipulation, in this case guidance, will be povided to the dominant hand while the use holds the stylus. This follows the two-handed inteaction wok defined by Guiad [4]. It is ou goal to guide the use to the ROI even if this is not visible. The use, theefoe, can hide the fmri data. The use is also capable of tuning on and off the navigation aid. Some subjects may pefe the aid to be of a constant speed instead of inceasing when close to the ROI. An option whee the use can nomalize the vectos is also pesented. The maximum foce applied to the stylus is also configuable.

4 The system shows a low-esolution display of the thee othogonal and the abitay slices, as can be seen on the bottom of the Figue 3. The use is also pesented with a glyph of the foce vecto that is cuently applied to the stylus. The MRI and fmri data wee povided in DICOM fomat. The fmri was scanned while giving visual stimuli to the subject. Thee wee 191 images of each set, each at a 256x256 esolution. The esult was volume size of 191x256x256 cubic voxels. These dimensions coespond to the thee volumes, the base MRI, the fmri and the FVM. Figue 3: An example of how the system looks. We highlight the main tools povided to the use. 5. Haptic Rendeing Methods To display the appopiate foce to the stylus, a FVM based on the fmri data, is geneated. Theefoe, the dimension of this Mesh is also 191x256x256. The FVM can be geneated unde vaious methods. Next, we pesent the two developed unde ou cuent implementation. Each FVM was geneated offline and saved to a file, since both methods equie consideable computational powe. We have to geneate the FVM fo each method of the fmri befoe using it in ou system. We conside a ROI as a goup of Points of Inteest (POI). Each POI coesponds to one voxel in the fmri volume. Both methods use an invese distance function to modify the magnitude of the foce. The magnitude of the esulting foce vecto needs to be mapped to an amount of foce that the PD can apply. We use a linea mapping to nomalize the maximum foce in the FVM to the maximum foce the PD can geneate. Altenatively splines and othe types of cuves could also be implemented Invese Distance Function The invese distance function on evey method was calculated by the following fomula: 1 g ( x) = 2 x By using this function, we ensue that the close the voxel to the POI, the stonge the foce of attaction will be Closest Point Method This method calculates each foce vecto by taking the POI closest to the voxel and calculating a vecto between these two. To detemine the poximity, we use the geometic distance. Once choosing the vecto, the magnitude will be modified to be equal to the nomalized vecto times the invese distance function descibed ealie. This method can also be defined with the following fomula: A F = g( d) d whee A = {min( V d = A ) V = Pi P; i = 0.. n} In this fomula, F is the foce vecto, g(x) is the invese distance function descibed ealie. Pi the ith point of inteest; P the cuent voxel at which we want to calculate the foce vecto, A is the vecto with the minimum magnitude of all the vectos fom P to the point of inteest Pi, and d is the magnitude of A. The value n is the numbe of points of inteest. In this method we assume all the points to have the same intensity, we use this as the analogous of mass. An example of how this method woks is given in Figue 4. Black cicles, labeled P1 to P4, ae sample POIs, and cicles in gay, labeled as A1 to A3, ae sample positions fo the pobe object. Position A1 is obviously close to P1 and theefoe points towads it. Given the line fomed by point P1 and P3, assume the x coodinate of position A2 is midway between the x coodinate of P1 and P3, but that the y coodinate of A2 is not aligned with these two points, but is highe up. By calculating the geometic distance, we find that A2 is pointing towads P2. As P1 and P3 ae collinea, let s assume the position A3 lies on this line and is midway between P1 and P3. When the position of the pobe happens to be an equal distance fom two o moe POIs, the system will use the fist calculated vecto, in this case, the vecto to P1.

5 Figue 4: Example of the behavio of vectos with the Closest Point Method Sum of Attactions Method The second method we povide is the Sum of Attactions. This method ceates an FVM in which each vecto is geneated by the following fomula: n Vi F = Ii g( di) di whee Vi = Pi P di = i Pi P F is the foce vecto we ae calculating; Ii the intensity at point i. Pi the ith point of inteest; P the cuent voxel at which we want to calculate the foce vecto, and di is the distance between the two points, Vi is the vecto between Pi and P, g(x) is the distance function descibed ealie. Again, the value n is the numbe of points of inteest. The esulting vecto will be detemined by the summation of the nomalized vecto fomed by the cuent voxel and evey POI, times the distance function evaluated on the magnitude of this vecto befoe nomalization, times the intensity of the point. Notice that this method is simila to a gavity model. The intensity of the point is analogous to the mass of the point In Figue 5 we pesent an example of how this system woks. We define a egion as those points in close poximity with each othe. The tem density is used fo the numbe of points in a egion. The diection of the vecto is not as obvious as in the pevious method. Point A3 is pointing towads the egion defined by P2, P3 and P4 because of its poximity and the fact that it is in a egion of high density. Point A2 is pointing towads this same egion even though it is close to P1, because the egion fomed by P2, P3 and P4 is dense than P1. Point A1, howeve, is not pointing towads P2, P3 and P4 even Figue 5: Example of the behavio of vectos with the Sum of Attactions Method. though its density is highe, but because, geometically, the distance to P1 is much shote and theefoe of moe influence than the density. 6. Poblems with the Cuent System The geneation of the FVM is a computationally expensive task, which is why it has to be pecomputed. Fo this eason, ou system cannot handle dynamic changes in the data. Also, since ou solution woks on discete space epesenting constant infomation, it is pone to aliasing atifacts. Anothe impotant issue is the diection of the ROI when it lies on a paallel plane above o below the displayed plane in the data set. We ae able to compute the diection to which the plane should be moved, howeve, an automatic option should be implemented. Two solutions ae in pogess ight now. One solution will povide feedback to the use of "climbing up a hill" when the plane is somewhee in the data stack "above" the displayed plane, and "climbing down a valley" when the plane is "below." These types of haptic feedback have aleady been tested on the PD [14]. Anothe appoach to solving this poblem is by automatically moving the cuent plane towads the desied diection. Both of these solutions have staightfowad implementations, howeve, the authos have decided to leave this as a futue wok. 7. Summay and Futue Wok We have pesented a technique fo navigation on lage sets of volumetic data using haptic feedback. Ou appoach povides a two-dimensional solution fo the exploation of thee dimensional data sets by using the Poactive Desk haptic feedback device. The cuent eseach is one of the fist exploiting the advantages of this new device. One of the paticula contibutions of this wok is the use of the use's natual undestanding of infomation displayed on a flat suface.

6 To conduct subject studies to pove the usability of ou system is an obvious next step. The fist study in mind is to test the advantages of guiding the use ove a flat suface towads a defined taget. In this test we will also compae the two methods we used to ceate the FVM. A second study will include the full functionality of the system while moving the cutting plane in a thee dimensional space. Analysis of the esults of these studies will help us confim whethe ou system povides a significant aid to the poblem of localizing infomation. The manipulation of the thee-dimensional data is cuently done with a tackball. Howeve, bette appoaches fo manipulation such as those by Hinckley et al. [6] o Föhlich and Plate [3] may be implemented in the next vesion. Moe techniques fo geneating the FVM will be eseached in the next vesion of the poject. The volume is cuently examined pixel by pixel. Using an Octee may educe stoage equiements in empty aeas, as noted by Avila et al. [2]. A custom-sized pobe as that noted by Petesik et al. [12] may be tested instead of the point-based one we use. Ou cuent system povides a way of guiding the use to a ROI, howeve, no discussion was pesented on haptically displaying these egions once eached. Thee ae many appoaches in solving this poblem, such as that of Noma and Iwata [11]. We can implement othe well-known techniques once ou guidance appoach is validated. 8. Acknowledgements This eseach was suppoted in pat by the National Institute of Infomation and Communications Technology. Refeences [1] Aliakseyeu D., Subamanian S., Matens J. B., Rautebeg M., Inteaction Techniques fo Navigation though and Manipulation of 2D and 3D Data, In Eight Euogaphics Wokshop on Vitual Envionments, 2002, pp [2] Avila R. S., Sobieajski L. M., A Haptic Inteaction Method fo Volume Visualization, In Poceedings of IEEE Visualization, 1996, pp [3] Föhlich B. and Plate J., The Cubic Mouse A New Device fo Thee-Dimensional Input In Poceedings of SIGCHI Confeence on Human Factos in Computing Systems, 2000, pp [4] Guiad Y., Asymmetic Division of Labo in Human Skilled Bimanual Action: The Kinematic Chain as a Model, Jounal of Moto Behaviou, 19(4): [5] Haan G. de, Koutek M., Post F. H., Towads Intuitive Exploation Tools fo Data Visualization in VR, In Poceedings of the ACM symposium in Vitual Reality Softwae and Technology, 2002, pp [6] Hinckley K., Pausch R., Goble J. C., Kassel N. F., Passive Real-Wold Inteface Pops fo Neuosugical Visualization, In Gaviel Salvendy, edito, Poceedings of ACM CHI 94 Confeence on Human Factos in Computing Systems, 1994, pp [7] Ikits M., Bedeson J. D., Hansen C. D., Johnson C. R., A Constaint-Based Technique fo Haptic Volume Exploation, In Visualization 2003, 2003, pp [8] Iwata H. and Noma H., Volume Haptization, In Poceedings IEEE Vitual Reality Annual Intenational Symposium, 1993 pp [9] Lundin K., Ynneman A., Gudmundsson B., Poxy-based Haptic Feedback fom Volumetic Density Data, In EuoHaptics, 2002, pp [10] Noma H., Yoshida S., Yanagida Y., Tetsutani N., The Poactive Desk : A New Foce Display System fo a Digital Desk Using a 2-DOF Linea Induction Moto, In Pesence: Teleopeatos & Vitual Envionments, Apil 2004, vol. 13, no. 2, pp [11] Noma H and Iwata H., Pesentation of Multiple Dimensional data by 6.D.O.F. foce display, In Poceedings of IEEE/RSJ Intenational Confeence on Intelligent Robots and Systems, 1993, pp [12] Petesik A., Benhad P., Ulf T., Kal H. H., Haptic Rendeing of Volumetic Anatomic Models at Sub-Voxel Resolution, Symposium on Haptic Intefaces fo Vitual Envionment and Teleopeato Systems, 2002, pp [13] Reimesdahl T. van, Bley F., Kuhlen T., Bischof C.H., Haptic Rendeing Techniques fo the Inteactive Exploation of CFD Datasets in Vitual Envionments, In Poceedings of the Wokshop on Vitual Envionments, 2003, pp [14] Yoshida S., Kuumisawa J. Noma H., Tetsutani N., Hosaka K., "Sumi-Nagashi: Ceation of New Style Media At with Haptic Digital Colos", ACM Multimedia 2004, , pp