Fusion of Data from Head-Mounted and Fixed Sensors 1

Transcription

1 Frst Internatonal Workshop on Augmented Realty, Nov., 998, San Francsco. Fuson of Data from Head-ounted and Fxed Sensors Abstract Wllam A. Hoff Engneerng Dvson, Colorado School of nes Golden, Colorado A methodology s developed to explctly fuse sensor data from a combnaton of fxed and head-mounted sensors, n order to mprove the regstraton of objects n an augmented realty system. he methodology was appled to the analyss of an actual expermental augmented realty system, ncorporatng an optcal see-through head-mounted dsplay, a head-mounted CCD camera, and a fxed optcal trackng sensor. he purpose of the sensng system was to determne the poston and orentaton (pose) of a movable object wth respect to the head-mounted dsplay. A typcal confguraton was analyzed and t was shown that the hybrd system produces a pose estmate that s sgnfcantly more accurate than that produced by ether sensor actng alone. Usng only the fxed sensor, the maxmum translatonal error n the locaton of an object wth respect to the head-mounted dsplay n any drecton was 8.23 mm (correspondng to a 97% confdence nterval). Usng only the head-mounted sensor, the maxmum translatonal error n any drecton was 9.9 mm. By combnng data from the two sensors, the maxmum translatonal error was reduced to.47 mm. In order to fuse the pose estmates, the uncertantes are explctly calculated, n the form of covarance matrces. A capablty was also developed to vsualze the uncertantes as 3D ellpsods. Introducton Where s an object of nterest wth respect to the user s head? In augmented realty systems that use head-mounted dsplays (HD s), knowng the relatve poston and orentaton (pose) between object and head s crucal n order to dsplay a vrtual object that s algned wth the real object. If the estmated pose of the object s naccurate, the real and vrtual objects may not be regstered correctly. Regstraton naccuracy s one of the most mp ortant problems lmtng augmented realty applcatons today []. hs work was supported by a grant from Johnson & Johnson rofessonal, Inc.

2 o determne the pose of an object wth respect to the user s head, trackng sensors are necessary. Optcal sensors use cameras or photo-effect sensors to track optcal targets, such as lght emttng dodes (LED s) or passve fducal markngs [2] [3] [4]. Usng two or more sensors (stereo vson), the threedmensonal (3D) poston of a target pont can be determned drectly va trangulaton. he accuracy of locatng the pont s mproved by ncreasng the separaton (baselne) between the sensors. he full sx degree-of-freedom (DOF) pose of a rgd body can be determned by measurng three or more target ponts on the body, assumng the geometry of the ponts on the body s known. hs procedure s known as the absolute orentaton problem n the photogrammetry lterature. Alternatvely, a sngle sensor can be used to measure the 2D (mage) locatons of three or more target ponts on a rgd body. If the geometry of the ponts s known, the full 6 DOF pose of the rgd body can be estmated, by a procedure s known as exteror orentaton [5]. One ssue s where to locate the sensor and target. One possblty s to mount the sensor at a fxed known locaton n the envronment, and put targets on both the HD and on the object of nterest (a confguraton called outsde n [3]). We measure the pose of the HD wth respect to the sensor, and the pose of the object wth respect to the sensor, and derve the relatve pose of the object wth respect to the HD. Another possblty s to mount the sensor on the HD, and the target on the object of nterest (a confguraton called nsde out ). We measure the pose of the object wth respect to the sensor, and use the known sensor-to-hd pose to derve the relatve pose of the object wth respect to the HD. Both approaches have been tred n the past, and each has advantages and dsadvantages. Wth a fxed sensor (outsde-n approach), there s no lmtaton on sze and weght of the sensor. ultple cameras can be used, wth a large baselne, to acheve hghly accurate 3D measurements va trangulaton. For example, commercal optcal measurement systems such as Northern Dgtal s Optotrak have baselnes of approxmately meter and are able to measure the 3-D postons of LED markers to an accuracy of less than 0.25 mm. he orentaton and poston of a target pattern s then derved from the ndvdual pont postons. A dsadvantage wth ths approach s that head orentaton must be nferred ndrectly from the pont postons. he nsde-out approach has good regstraton accuracy, because a slght rotaton of a head-mounted camera causes a large shft of a fxed target n the mage. However, a dsadvantage of ths approach s that one t s mpossble to put multple cameras wth a large baselne separaton on the head. Ether a small baselne separaton must be used, or alternatvely a sngle camera can be used wth the exteror orentaton algorthm. Ether method gves rse to large translaton errors along the lne of sght of the cameras. 2

3 A queston arses s t possble to fuse the data from a head-mounted sensor and a fxed sensor, and derve a more accurate estmate of object-to-hd pose? If the data from these two types of sensors are complementary, then the resultng pose can be much more accurate than that from each sensor used alone. Effectvely, we can create a hybrd system that combnes the nsde-out and outsde-n approaches. hs paper descrbes a methodology to explctly compute uncertantes of pose estmates, propagate these uncertantes from one coordnate system to another, and fuse pose estmates from multple sensors. he contrbuton of ths work s the applcaton of ths methodology to the regstraton problem n augmented realty. It s shown that a hybrd sensng system, combnng both head-mounted and fxed sensors can mprove regstraton accuracy. 2 Background on ose Estmaton 2. Representaton of ose he notaton n ths secton follows that of Crag [6]. he pose of a rgd body {A} wth respect to another coordnate system {B} can be represented by a sx B B B B element vector ( ) A x Aorg, yaorg, z Aorg, α, β, γ B B B B ( x, y, z ) Aorg Aorg, where s the orgn of frame {A} n frame {B}, and Aorg Aorg (α,β,γ) are the angles of rotaton of {A} about the (z,y,x) axes of {B}. An alternatve representaton of orentaton s to use three elements of a quaternon; the converson between xyz angles and quaternons s straghtforward. Equvalently, pose can be represented by a 4x4 homogeneous transformaton matrx: where B A B B A R Aorg H () 0 B A R s a 3x3 rotaton matrx. In ths paper, we shall use the letter to desgnate a sx-element pose vector and the letter H to desgnate the equvalent 4x4 homogeneous transformaton matrx. Homogeneous transformatons are a convenent and elegant representaton. A A A A Gven a homogeneous pont ( x, y, z, ), represented n coordnate system {A}, t may be transformed to coordnate system {B} wth a smple B B A matrx multplcaton H. he homogeneous matrx representng the A pose of frame {B} wth respect to frame {A} s just the nverse of the pose of A {A} wth respect to {B};.e., H H. Fnally, f we know the pose of {A} B B A wth respect to {B}, and the pose of {B} wth respect to {C}, then the pose of 3

4 {A} wth respect to {C} s easly gven by the matrx multplcaton C C B A H B H AH. 2.2 ose Estmaton Algorthms he problem of determnng the pose of a rgd body, gven an mage from a sngle camera, s called the exteror orentaton problem n photogrammetry. Specfcally, we are gven a set of 3D known ponts on the object (n the coordnate frame of the object), and the correspondng set of 2D measured mage ponts from the camera, whch are the perspectve projectons of the 3D ponts. he nternal parameters of the camera (focal length, prncpal pont, etc.) are known. he goal s to fnd the pose of the object wth respect to the camera, cam obj. here are many solutons to the problem; n ths work we used the algorthm descrbed by Haralck [5], whch uses an teratve non-lnear least squares method. he algorthm effectvely mnmzes the squared error between the measured 2D pont locatons and the predcted 2D pont locatons. he problem of determnng the pose of a rgd body, gven a set of 3D pont measurements, s called the absolute orentaton problem n photogrammetry. hese 3D pont measurements may have been obtaned from a prevous trangulaton process, usng a sensor consstng of multple cameras. Specfcally, we are gven a set of 3D known ponts on the object { obj }, and the correspondng set of 3D measured ponts from the sensor { sen }. he goal s to fnd the pose of the object wth respect to the sensor, sen obj. here are many solutons to the problem; n ths work we used the algorthm descrbed by Horn [7] whch uses a quaternon-based method. 3 Determnaton and anpulaton of ose Uncertanty Gven that we have estmated the pose of an object, usng one of the methods above, what s the uncertanty of the pose estmate? Knowng the uncertanty s crtcal to fusng measurements from multple sensors. We can represent the uncertanty of a sx-element pose vector, by a 6x6 covarance matrx C E, whch s the expectaton of the square of the dfference ( ) between the estmate and the true vector. hs secton descrbes methods to estmate the covarance matrx of a pose, transform the covarance matrx from one coordnate frame to another, and combne two pose estmates. 4

5 5 3. Computaton of Covarance Assume that we have N measured data ponts from the sensor {, 2,..., N }, and the correspondng ponts on the object {Q, Q 2,..., Q N }. he object ponts Q are 3D; the data ponts are ether 3D (n the case of 3D-to-3D pose estmaton) or 2D (n the case of 2D-to-3D pose estmaton). We assume that the nose n each measured data pont s ndependent, and the nose dstrbuton of each pont s gven by a covarance matrx C. Let H(Q, ) be the functon whch transforms object ponts nto data ponts. In the case of 3D-to-3D pose estmaton, ths s just a multplcaton of Q by the correspondng homogeneous transformaton matrx. In the case of 2Dto-3D pose estmaton, the functon s composed of a transformaton followed by a perspectve projecton. An algorthm that solves for est mnmzes the sum of the squared errors. Assume that have we solved for est usng the approprate algorthm (.e., 2D-to-3D or 3D-to-3D). We then lnearze the equaton about the estmated soluton est: ( ) ( ) H Q H Q H Q est est est + + +,,, (2) Snce H(Q, est ), the equaton reduces to H Q est, (3) where s the Jacoban of H, evaluated at (Q, est ). Combnng all the measurement equatons, we get the matrx equaton: N N (4) Solvng for, we get ( ). he covarance matrx of s gven by the expectaton of the outer product: ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) C C E E E C 0 0 L O L (5)

6 Note that we have assumed that the errors n the data ponts are ndependent;.e., E( j )0, for j. (If the errors n dfferent data ponts are actually correlated, our smplfed assumpton could result n an underestmate of the actual covarance matrx.) he above analyss was verfed wth onte Carlo smulatons, usng both the 3D-to-3D algorthm and the 2D-to-3D algorthm. 3.2 Interpretaton of Covarance A useful nterpretaton of the covarance matrx can be obtaned by assumng that the errors are jontly Gaussan. he jont probablty densty for N- dmensonal error vector s [8]: p N 2 ( ) ( C ) 2 exp ( C ) 2π (6) If we look at surfaces of constant probablty, the argument of the exponent s a 2 constant, gven by the relaton C z. hs s the equaton of an ellpsod n N dmensons. For a gven value of z, the cumulatve probablty of an error vector beng nsde the ellpsod s. For N3 dmensons, the ellpsod defned by z3 corresponds to a cumulatve probablty of approxmately 97% 2. For a sx-dmensonal pose, the covarance matrx C s 6x6, and the correspondng ellpsod s sx dmensonal (whch s dffcult to vsualze). However, we can select only the 3D translatonal component of the pose, and look at the covarance matrx correspondng to t. Specfcally, let Z(x,y,z) be the translatonal porton of the pose vector (x,y,z,α,β,γ). We obtan Z from usng the equaton Z, where s the matrx (7) he covarance matrx for Z s gven by C Z C (whch s just the upper left 3x3 submatrx of C ). We can then vsualze the three dmensonal ellpsod correspondng to C Z. 3.3 ransformaton of Covarance We can transform a covarance matrx from one coordnate frame to another. Assume that we have a sx-element pose vector and ts assocated covarance matrx C. Assume that we apply a transformaton, represented by a sxelement vector W, to to create a new pose Y. Denote Y g(, W). A aylor 2 he exact formula for the cumulatve probablty n N dmensons s 2 e d [8]. 2N / 2 N ( ) N Γ N 2+ z 2 6

7 seres expanson yelds Y J, where J ( g/ ). he covarance matrx C Y s found by: Y [ ] J E( ) J J C J ( YY ) E ( J )( J ) C E (8) A varaton on ths method s to assume that the transformaton W also has an assocated covarance matrx C W. In ths case, the covarance matrx C Y s: C J C J + J C J (9) Y where J ( g/ ) and J W ( g/ W ). he above analyss was verfed wth onte Carlo smulatons, usng both the 3D-to-3D algorthm and the 2D-to-3D algorthm. 3.4 Combnng ose Estmates wo vector quanttes may be fused by averagng them, weghted by ther covarance matrces. Let, 2 be two N-dmensonal vectors, and C, C 2 be ther NxN covarance matrces. Assumng and 2 are uncorrelated, then the combned estmate and the combned covarance matrx C may be found by the followng equatons 3 : C 2 ( C + C2 ) + C ( C + C2 ) ( C + C2 ) C C C2 herefore, ths s a method of sensor fuson n the hybrd augmented realty system. If the pose of an object wth respect to the HD can be estmated usng data from the head-mounted sensor, and the same pose can be estmated usng data from the fxed sensor, then a combned estmate can be produced usng Equaton 0. When combnng pose estmates, we use a quaternon-based representaton of orentaton, rather than xyz angles or Euler angles. he reason s that xyz angles have a problem for orentatons where one angle s close to 80. In ths case, one of the pose vectors may have a value for the angle close to +80, and the other vector may have a value close to -80. Even though the two vectors represent very smlar orentatons, the combned vector would represent a wldly dfferent orentaton. Quaternons do not have ths problem. 4 Experments he methodology descrbed n the prevous sectons was appled to an actual expermental augmented realty system developed n our lab. he purpose of W W W 2 (0) 3 hese equatons can be derved from the dscrete Kalman flter update equatons, usng as the a pror estmate, 2 as the measurement, and as the a posteror estmate. 7

8 the system s to dsplay a graphcal overlay on an HD, such that the overlay s regstered to a movable object n the scene. Only quas-statc regstraton s consdered n ths paper; that s, objects are statonary when vewed, but can freely be moved. he system ncorporates both head-mounted and fxed sensors. he hybrd system was developed for a surgcal ad applcaton, but ts capabltes are such that t could be used n many other applcatons. he frst sub-secton below descrbes the expermental setup, ncludng the sensor and dsplay characterstcs. Addtonal detals of the system are descrbed n [9]. In the second sub-secton, the task tself s descrbed. Fnally, an analyss of regstraton accuracy s performed. 4. Descrpton of Expermental AR System he prototype augmented realty system ncorporates a see-through HD (Vrtual I-O -glasses ) mounted on a helmet (Fgure, left). A CCD camera wth a feld of vew of 44 degrees s also mounted on the helmet. he NSCformat vdeo sgnal from the camera s transmtted to a C through a cable tether, whch dgtzes and processes the mage. An optcal target s affxed to the object of nterest (Fgure, rght). For ths work, we used a pattern of 5 green LED s, n a rectangular planar confguraton. he dstnctve geometrc pattern of the LED s enables the correspondence to be easly determned [9]. he C performs low-level mage processng to extract the mage locatons of the LED targets. he nose n the 2D measured mage pont locatons was assumed to be sotropc, wth an estmated standard devaton of 0.5 pxels. ose estmaton s done usng a 2D-to-3D algorthm. he throughput currently acheved wth the system s approxmately 8.3 Hz. Our fxed sensor was an optcal measurement system (Northern Dgtal Optotrak 3020) fastened to one wall of the laboratory. he sensor conssts of three lnear array CCD cameras. An optcal target, consstng of a set of sx nfrared LED's, s fastened to each object of nterest. he cameras detect each LED and calculate (va trangulaton) ts 3D locaton wth respect to the sensor. From the resultng set of 3D pont postons on a target body, the controller also calculates the pose of the body wth respect to the sensor. For 8 target ponts, we measured an update rate of approxmately 4 Hz. Infrared LED s were also placed on the helmet, to form an optcal target. A set of 6 LED s were mounted n a sem-crcular rng around the front half of the helmet (Fgure, left). ypcally, only 4 LED s were vsble at any one tme. he measurement nose was assumed to be sotropc, wth σ 0.5 mm. 8

9 4.2 Descrpton of ask he hybrd augmented realty system was developed for a surgcal ad applcaton; specfcally, total hp jont replacement. he purpose of the augmented realty system s to track a hp mplant and dsplay a graphcal overlay on the HD, that s regstered to the mplant. Optcal targets were attached to the mplant to enable sensor trackng, as shown n Fgure (rght). Separate LED targets were used for the head-mounted and fxed sensors. he prncpal coordnate frames used n the system are lsted and descrbed n able, and depcted schematcally n Fgure 2. Even though ths fgure shows all frames as co-planar, the transformatons between frames are actually fully sx-dmensonal (.e., three translatonal and three rotatonal components). o ad n vsualzng these coordnate frames, a 3D graphcal dsplay system was developed usng a Slcon Graphcs workstaton and the Open Inventor graphcs package. Fgure 3 (left) shows a smplfed representaton of the coordnate frames on the head: the HD, the HD target, and the headmounted camera. hese coordnate frames are rgdly mounted wth respect to each other on the helmet. Fgure 3 (rght) shows a smplfed representaton of the coordnate frames attached to the mplant: the mplant, the mplant target, and the camera target. hese coordnate frames are also rgdly mounted wth respect to each other. (he real helmet and mplant assembles were shown n Fgure.) he coordnate axes of all frames are also shown. Fgure 4 (left) shows the entre room scene, consstng of the fxed sensor on the back wall, the observer wth the HD, and the patent on the table wth the hp mplant. Fgure 4 (rght) shows a 3D vsualzaton of the same scene. 4.3 Analyss of Regstraton Accuracy A smulaton was mp lemented, usng the software applcaton athematca, to estmate the accuracy of the derved mplant-to-hd pose. he processng conssts of three man steps. Frst, an estmate of mplant-to-hd pose s derved usng data obtaned from the Optotrak (fxed) sensor alone. Second, an estmate of mplant-to-hd pose s derved usng data obtaned from the headmounted camera alone. Fnally, the two estmates are fused to produce a sngle, more accurate estmate. hese steps are descrbed n detal below ose Estmaton from Fxed Sensor Usng data from the fxed sensor (Optotrak), we estmated the pose of the HD Optotrak target ( H ) wth respect to the sensor, usng the 3D-to-3D algorthm Hmdarg descrbed earler. From the estmated error n each 3D pont measurement (0.5 mm), the covarance matrx of the resultng pose was determned. Usng the 9

10 known pose of the HD wth respect to the HD target ( Hmdarg Hmd H ), the pose of the HD wth respect to the sensor was estmated, usng the equaton Optotrak Optotrak Hmdarg Hmd H HmdargH HmdH. he covarance matrx of the resultng pose was also estmated. he ellpsods correspondng to the uncertantes n the translatonal components of the poses are shown n Fgure 5 (left). In all fgures n ths paper, the ellpsods are drawn correspondng to a normalzed dstance of z3;.e., correspondng to a cumulatve probablty of 97%. However, durng renderng the ellpsods are scaled up by a factor of 0 n order to make them more easly vsble. he major axs of the small ellpsod n Fgure 5 (left) s actually 0.32 mm; that of the larger ellpsod s.84 mm. Next, the fxed sensor estmated the pose of the mplant target ( Optotrak Imparg H ) wth respect to the sensor, along wth the correspondng covarance matrx. Usng the Imparg known pose of the mplant wth respect to the mplant target ( Implant H ), the pose of the mplant wth respect to the sensor was estmated, usng Optotrak Optotrak Imparg Implant H ImpargH ImplantH, along wth ts covarance matrx. Fnally, the pose of the mplant wth respect to the HD was estmated va. Hmd H ( opto ) Hmd Optotrak Implant OptotrakH ImplantH. he covarance matrx of ths pose was estmated usng Equaton 9. he correspondng ellpsod s shown n Fgure 5 (rght). he major axs of ths ellpsod s 8.23 mm. Note that the shape of ths ellpsod s elongated n the plane perpendcular to the lne of sght, due to the orentaton uncertanty n the HD ose Estmaton Usng Head-ounted Sensor Usng data from the head-mounted camera, we estmated the pose of the camera Camera target ( H ) wth respect to the camera, usng the 2D-to-3D algorthm Camarg descrbed earler. From the estmated error n each 2D-pont measurement (0.5 pxel), the covarance matrx of the resultng pose was determned. hen, usng Camarg the known pose of the mplant wth respect to the camera target ( Implant H ), the pose of the mplant wth respect to the camera was estmated, va Camera Camera Camarg Implant H CamargH ImplantH. he covarance matrx of the resultng pose was also estmated. he ellpsods correspondng to the translatonal uncertantes are shown n Fgure 6 (left). he major axs of the ellpsod correspondng to Camera Camarg H s 24.6 mm. he major axs of the ellpsod correspondng to the derved pose, Camera Implant H, s 9.9 mm. Note the large uncertanty of Camera Camarg H along the lne of sght to the camera, and very small uncertanty perpendcular to the lne of sght. hs s typcal of poses 0

11 that are estmated usng the 2D-to-3D method. Intutvely, ths may be explaned as follows. A small translaton of the object parallel to the mage plane results n an easly measurable change n the mage, meanng that the uncertanty of translaton s small n ths plane. However, a small translaton of the object perpendcular to the mage plane generates only a very small mage dsplacement, meanng that the uncertanty of translaton s large n ths drecton. Next, the pose of the mplant wth respect to the HD s estmated, va. Hmd H ( cam ) Hmd Camera Implant CameraH ImplantH. he covarance matrx of ths pose was estmated usng Equaton 9. he correspondng ellpsod s shown n Fgure 6 (rght). he major axs of ths ellpsod s 9.9 mm Fuson of Data from Fxed and Head-ounted Sensors he two pose estmates, whch were derved from the fxed and head-mounted sensors, can now be fused. Usng Equaton 0, we produce a combned estmate of the mplant-to-hd pose, along wth the covarance matrx. he ellpsods correspondng to the three poses, Hmd (opto) Hmd (cam) Implant H, Implant H, and are shown n Fgure 7. Note that the large ellpsods, correspondng to Hmd (opto) Hmd (cam) Implant H and Implant H correspondng to the combned pose,, are nearly orthogonal. he ellpsod Hmd (hybrd) Implant H Hmd (hybrd) Implant H, s much smaller and s contaned wthn the ntersecton volume of the larger ellpsods. he rght mage of Fgure 7 s a wre-frame renderng of the ellpsods, whch allows the smaller nteror ellpsod to be seen. he major axs correspondng to the uncertanty of the combned pose s only.47 mm. 5 Summary hs paper has developed a methodology to explctly fuse sensor data from a combnaton of fxed and head-mounted sensors, n order to mprove the regstraton of objects wth respect to a HD. he methodology was appled to an actual expermental augmented realty system. A typcal confguraton was analyzed and t was shown that the hybrd system produces a pose estmate that s sgnfcantly more accurate than that produced by ether sensor actng alone. 6 Acknowledgments he author would lke to thank Dr. yrone Vncent for many helpful dscussons, Kho Nguyen for mplementng many of the components of the

12 expermental augmented realty system, and the anonymous revewers for ther helpful comments. 7 References [] R.. Azuma, A Survey of Augmented Realty, resence, Vol. 6, No. 4, pp , 997. [2] R. Azuma and G. Bshop, Improvng statc and dynamc regstraton n an optcal see- through HD, roc. of 2st Internatonal SIGGRAH Conference, AC, pp , 994. [3] J.-F. Wang, et al, rackng a Head-ounted Dsplay n a Room-Szed Envronment wth Head-ounted Cameras, roc. of Helmet-ounted Dsplays II, Vol. 290, SIE, pp , 990. [4] D. Km, S. W. Rchards, and.. Caudell, An optcal tracker for augmented realty and wearable computers, roc. of IEEE 997 Annual Internatonal Symposum on Vrtual Realty, pp , 997. [5] R. Haralck and L. Shapro, Computer and Robot Vson, Addson- Wesley Inc, 993. [6] J. Crag, Introducton to Robotcs, echancs, and Control, 2nd ed., Addson Wesley, 990. [7] B. K.. Horn, Closed-form soluton of absolute orentaton usng unt quaternons, J. Optcal Soc. of Amerca, Vol. 4, No. 4, pp , 987. [8] H. L. V. rees, Detecton, Estmaton, and odulaton heory, New York, Wley, 968. [9] W. A. Hoff,. Lyon, and K. Nguyen, Computer Vson-Based Regstraton echnques for Augmented Realty, roc. of Intellgent Robots and Computer Vson V, Vol. 2904, SIE, pp , 996. able rncpal coordnate frames n the system. Frame HD Implant HD target Camera Implant target Camera target Descrpton Centered at left eyepece of dsplay Centered on mplant component Optcal target mounted on helmet, tracked by fxed sensor Camera mounted on helmet Optcal target attached to mplant, tracked by fxed sensor Optcal target attached to mplant, tracked by head-mounted camera 2

13 Cameras LED s HD Fgure (Left) rototype augmented realty system. Only one of the cameras was used. (Rght) Fve green LED's (top surface) form the optcal target for the head mounted camera. Sx nfrared LED s (front surface) form an optcal target for the Optotrak sensor. Both targets are mounted on a box, whch s attached to a hp mplant component. {HD target} {Camera} {Optotrak} {HD} Legend: {Coordnate frame} Known transformaton {Implant} easured transformaton Derved transformaton {Implant target} {Camera target} Fgure 2 he prncpal coordnate frames n the system are shown, along wth the transformatons between them. 3

14 HD optcal target Camera Implant Camera target HD Implant target Fgure 3 he coordnate frames on the head (left) and on the mplant (rght). Fgure 4 A vsualzaton of the entre scene, showng the fxed sensor on the wall, the HD, and the hp mplant. (Left) he real scene. (Rght) A 3D vsualzaton. Uncertanty ellpsods Uncertanty ellpsod Fgure 5 Uncertantes of poses derved from fxed sensor: (Left) HD target (small ellpsod) and HD (large ellpsod). (Rght) Implant wth repsect to HD. 4

15 Uncertanty ellpsods Uncertanty ellpsod Fgure 6 Uncertantes of poses derved from head camera: (Left) Camera target (long narrow ellpsod) and mplant wth respect to camera (wde ellpsod). (Rght) Implant wth respect to HD. Uncertanty ellpsod (derved from fxed sensor) Uncertanty ellpsod (derved from headmounted sensor) Fgure 7 (Left) hs fgure depcts the fuson of the data. Note that the ellpsods from the fxed sensor and the head-mounted sensor are nearly orthogonal. he ellpsod correspondng to the resultng pose estmate s much smaller and s contaned n the volume of ntersecton. (Rght) hs wre-frame renderng of the uncertanty ellpsods allows the smaller (combned estmate) ellpsod to be seen, whch s contaned n the ntersecton of the two larger ellpsods. 5