Low Latency Rendering on Pixel-Planes 5. Jonathan Cohen Marc Olano University of North Carolina at Chapel Hill

Transcription

1 Low Latency Renderng on Pxel-Planes 5 Jonathan Cohen Marc Olano Unversty of North Carolna at Chapel Hll

2 2 Abstract In ths paper we present a renderng system called Slats, whch renders trangle data sets wth only a sngle feld tme (16.7 ms) of latency n the renderng ppelne. We measure ths renderng latency from the tme Slats begns transformng the data set nto screen coordnates (usng the most recent trackng nformaton) to the tme the dsplay devces begn to scan the pxel colors from the frame buffers onto the screens. The Slats ppelne runs on Pxel-Planes 5, though we hope that the deas we present and the compromses we make wll be useful to desgners of low latency ppelnes on other platforms. We begn wth an ntroducton to the latency problem, ncludng some dscusson of future see-through head-mounted dsplay applcatons requrng low latency renderng. Next, we present the prevous work n low latency renderng. We then present a bref, but necessary, hardware overvew, leadng nto a dscusson of the renderng ppelnes runnng on ths hardware. We descrbe and analyze both the standard Renderng Control ppelne and the Slats low latency ppelne. We follow ths wth a secton descrbng some problems we encountered and conclude by mentonng future work. Introducton Computer graphcs, at frst a statc medum for vsualzng large volumes of data, s becomng more and more a medum for dynamc nteractons. Hghly parallel, specalzed graphcs archtectures have empowered graphcs platforms wth the ablty not only to dsplay varous vews of vrtual objects or worlds, but to do so quckly enough to convey the sensaton of moton. Our vsual systems buld on our knowledge of an object accordng to the way t moves, or appears to move as we change our vew of t. Performance of these current graphcs systems s commonly measured n terms of the number of trangles rendered per second or n terms of the number of complete frames rendered per second. Whle these measures are the most common, they don t tell the whole story. Latency, whch measures the start to fnsh tme of an operaton such as drawng a sngle frame, s an often neglected measure of graphcs performance. For some current modes of nteracton, lke manpulatng a 3D object wth a joystck, ths measure of responsveness may not be mportant. But for emergng modes of natural nteracton, latency s a crtcal measure. Head-mounted dsplays (HMDs) attempt such a natural mode of nteracton. The user s head poston and orentaton are tracked to determne what vewpont to use n generatng a stereo par of mages for the eye dsplays. The hgher the total latency n such a vsualzaton system, the more the world seems to lag behnd the user s head motons. The effect of ths lag s a hgh vscosty world.

3 3 The effect of latency s even more notceable wth see-through HMDs. Such dsplays supermpose computer generated objects on the user s vew of the physcal world. The lag becomes obvous n ths stuaton because the real world moves wthout lag, whle the vrtual objects shft n poston durng the lag tme, catchng up to ther proper postons when the user stops movng. Ths swmmng of the vrtual objects not only detracts from the desred lluson of the objects physcal presence, but also hnders any effort to use ths technology for real applcatons. Future applcatons of see-through HMDs wll augment the real world wth computer generated mages to enable us to see otherwse nvsble data and to take acton based on ths addtonal vsual nformaton. For nstance, ths augmented realty could help an archtect to vsualze buldng modfcatons before fnalzng the modfcaton plan. Or t could present 3D nstructons to gude the performance of complex 3D tasks [FEIN93], such as repars to a photocopy machne or even a jet engne. Ths technology could even grant computer smulated x-ray vson to doctors, allowng them to see n 3D the fetus nsde a pregnant woman or the tumor nsde a cancer patent. Current research nto the use of see-through HMDs by obstetrcans to vsualze 3D ultrasound data ndcates the need for lower latency vsualzaton systems [BAJU92]. The use of see-through HMDs for assstng surgcal procedures s unthnkable untl we make sgnfcant advances n the area of low latency graphcs systems. A possble soluton to ths lag problem s to use vdeo technques to cause the user s vew of the real world to lag n synchronzaton wth the vrtual world. Whle ths soluton may seem quck and easy, t doesn t really do anythng to fx the problem. You can t use your hands to nteract naturally wth a laggng world. Another soluton to the latency problem s to predct where the user s head wll be when the mage s fnally dsplayed. Ths technque, called predctve trackng, nvolves usng both recent trackng data and accurate knowledge of the system s total latency to make a best guess at the poston and orentaton of the user s head when the mage s dsplayed nsde the HMD. Predctve trackng alone, however, wll not eradcate the latency problem, because a large total latency n the trackng and mage generaton systems forces the predcton program to predct far nto the future, undermnng ts efforts to make accurate predctons. However, f we strve to reduce the total latency, we can hope to hde the effects of the remanng latency wth predctve trackng technques. We have desgned a renderng ppelne called Slats to be part of such a mnmal latency system. The renderng latency of Slats s only one feld tme (16.7 ms),

4 4 makng t deal for ths purpose. We measure ths renderng latency from the tme Slats begns transformng the data set nto screen coordnates (usng the most recent trackng nformaton) to the tme the dsplay devces begn to scan the pxel colors from the frame buffers onto the screens. Prevous Work The mnmzaton of latency s a relatvely new area of research. Whle most graphcs system desgners probably consder effects of ther archtectural choces on latency, t s not common to reduce latency at the expense of any sgnfcant throughput. But as people begn to wrte applcatons that use graphcs to augment realty rather than supersede t, latency wll become much more mportant. Matthew Regan and Ronald Pose are currently buldng the prototype for a hardware archtecture desgned to work wth vrtual realty systems [REGA93]. They elmnate latency for head rotatons by renderng a scene on the sx faces of a cube. The pxels are scanned out to the dsplay usng a beam racng technque nstead of beng read from a frame buffer. As a pxel s needed for dsplay, approprate memory locatons from the rendered cube faces are read. A head rotaton smply alters whch memory s accessed, and thus contrbutes nothng to the latency. The lookup process can take nto account dstorton functons wth no addtonal penalty, and smple ant-alasng s performed durng the beam racng process. Of course, a real user of an HMD always translates hs head n space n addton to rotatng t. Regan and Pose use object-space subdvson and mage composton to allow objects to be prortzed and re-rendered as necessary to accommodate translatons of the user s head. The mage may not always be correct f the renderng hardware cannot keep up, but the most mportant objects, whch nclude the closest ones, should be rendered n tme to keep ther postons accurate. Dstant objects, whch change very lttle as a result of head translatons, may not be re-rendered n tme, but should affect the user s percepton of the mage much less than the closer objects. The HMD research group at the Unversty of North Carolna s attemptng to reduce latency on currently exstng hardware. Ths s part of a multple-front attack on the more general problem of regsterng real objects wth computer generated ones [BISH93]. Latency s one of several factors we need to deal wth to reduce the swmmng of vrtual objects n the real world space. Mne has measured the total latency of our graphcs systems when confgured wth varous trackng systems [MINE93a]. He measures the latency at each stage of the ppelned system. These sorts of detaled measurements are necessary for fndng the largest sources of delay as well as for knowng how far nto the future we need to predct the user s head locaton to elmnate latency.

5 5 Mne and Bshop have also presented a technque called just-n-tme pxels, whch deals wth the placement of pxels on a scan-lne dsplay as a problem of temporal alasng [MINE93b]. Although the dsplay may take many mllseconds to refresh, the mage we see on the dsplay typcally represents only a sngle nstant n tme. When we see an object n moton on the dsplay, t appears dstorted because we see the hgher scan lnes before we see the lower ones, makng t seem as f the lower part of the object lags behnd the upper part. Avodance of ths dstorton entals generatng every pxel the way t should appear at the exact tme of ts dsplay. They also suggest approxmatng ths by applyng the proper transformaton for every scan-lne rather than for every pxel. The Slats system we wll present s much more practcal than just-n-tme pxels, but we can also see a theoretcal foundaton derved from that of just-n-tme pxels. It approxmates just-n-tme pxels by usng a new transformaton for every feld of the NTSC dsplay rather than for every pxel or every scan lne. However, ths s merely an after-the-fact observaton, because Slats was not developed wth just-n-tme pxels n mnd. Before gong nto the detals of Slats, we wll present some background on the hardware used and the renderng ppelne typcally used on ths hardware. Hardware We have mplemented Slats on Pxel-Planes 5 (Pxpl5) [FUCH89]. Ths s not to say that our approach does not apply to more readly avalable archtectures, but usng Pxpl5 gave us total control over the graphcs software, whch was all developed n-house. Because our goal was to acheve lower latency by modfyng the renderng ppelne, such low-level control was necessary. Before descrbng the standard renderng ppelne on Pxpl5 and our new renderng ppelne, we need to present some background on ths hardware platform. Pxel-Planes 5 uses parallelsm at both the transformaton and rasterzaton stages of the renderng process. Prmtves, such as trangles, are typcally generated on a host workstaton and sent va a rng network to a set of graphcs processors, where they are stored n local dsplay lsts. The graphcs processors traverse these dsplay lsts, transformng the prmtves from ther object-based coordnate systems to a screen-based coordnate system and generatng approprate renderng commands. The graphcs processors then send these commands over the rng to the renderers, whch perform rasterzaton and shadng. Fnally, the renderers send the resultng pxel values to a frame buffer, whch s synchronzed wth a vdeo dsplay for output.

6 6 32 bts, 20 MHz Rng Nodes 8 x 32 bts, 20 MHz Rng Network Graphcs Processor Workstaton Renderer Frame Buffer Rng Network Pxel-Planes 5 s rng network connects the host nterface, graphcs processors, renderers, and frame buffers. It provdes 8 communcatons channels, each capable of handlng 20 megawords/sec. Ths total of 160 megawords/sec s necessary to support the smultaneous transmsson of renderng commands, pxel values, and synchronzaton messages whle renderng at rates of 1M prmtves per second. Ths communcatons bandwdth s an mportant consderaton when determnng the feasblty of a new graphcs ppelne. Graphcs Processors The graphcs processors (GPs) are general purpose 64-bt Intel 860 processors, each wth 8 MB of DRAM. These processors run at 40 MHz. They functon as a MIMD set of nodes wth some synchronzaton to control ther states wth respect to that of the overall graphcs ppelne. The GPs typcally perform some transformatons on graphcs prmtves. Each works ndependently on the set of prmtves t has stored n ts RAM. We wll dscuss the nteracton of the GPs wth the rest of the system more thoroughly n the secton descrbng the renderng ppelne. Renderers The customzed renderers gve Pxel-Planes ts name. Each renderer s a SIMD array of 128x128 sngle-bt processors, wth each processor correspondng to a partcular pxel on the screen. A renderer rasterzes prmtves by evaluatng lnear expressons smultaneously for every pxel n the array. Some of these expressons represent lnes, whle others represent planes.

7 7 For example, a GP mght send the followng commands to a renderer to rasterze a trangle and mantan the renderer s Z-buffer for some screen regon. Frst, enable all the pxels. Next, evaluate a lnear equaton we call an edge expresson (though t actually represents the entre lne that ncludes some edge of the trangle). Dsable all the pxels sttng on the wrong sde of the lne, leavng all pxels on the correct sde (whch ncludes the nsde of the trangle) enabled. Evaluate two more edge expressons, leavng only the pxels nsde the trangle enabled. Now evaluate the Z-plane equaton, whch the GP determnes from the Z coordnates of the trangle s vertces. Next, compare the newly nterpolated Z value at each pxel wth the Z value of the closest prmtve at that pxel so far. If the new Z value s farther away from the vewer than the old one, the pxel dsables tself. For each pxel that s stll enabled, the new Z value replaces the old, and the pxel remans enabled for the nterpolaton of normal planes and color planes. The color, normal, and Z value computed are saved for the pxels n that prmtve. Any further shadng calculatons that make use of these values are deferred untl the end of the frame, when the renderers have fnshed executng the commands for all the prmtves. Ths deferred shadng technque allows the shadng to be done once per pxel nstead of once per pxel per prmtve [TEBB92]. When the renderer completes the shadng for the regon, t sends the fnal color values for that regon to a frame buffer. Frame Buffers Pxel-Planes 5 can operate wth both hgh-resoluton and NTSC-compatble frame buffers bult from vdeo RAMs. A hgh-resoluton frame buffer contans data for a 1280 x 1024 array of screen pxels. However, HMDs use NTSC vdeo, so we wll not consder the hgh-resoluton frame buffers further. An NTSC frame buffer contans data for a 640 x 480 resoluton dsplay, wth 24 bts of color data at each pxel. Each NTSC frame buffer s consdered a double buffer because there are two separate buffers representng the same set of pxels. Ths allows the graphcs software to wrte pxels nto one buffer wthout affectng the currently-dsplayed buffer. When the software s done wrtng some data, and a vertcal retrace occurs on the dsplay controller, the software swaps the two buffers, allowng the new mage to be scanned out. Each of these two buffers n an NTSC frame buffer contans two felds -- one contanng the odd lnes and one contanng the even lnes. Dvdng a frame nto felds s known as nterlaced dsplay; all the odd lnes are scanned out after one retrace, then the even lnes are scanned out after the next retrace. These retraces occur at 60 Hz. One addtonal complcaton s that we deal wth dual frame buffer stereo, usng an NTSC frame buffer for each eye. Ths means we can work wth up to eght

8 8 felds of data. The dsplay controllers for the left and rght eye dsplays are synchronzed, so the vertcal retrace comes at the same tme for both eyes. Renderng Ppelnes Now that we have covered the basc Pxel-Planes 5 hardware, we are ready to consder the renderng process. Renderng s typcally broken down nto several stages that can be ppelned to acheve hgher throughput on machnes wth enough hardware to execute the varous ppelne stages at the same tme. We wll begn by presentng the standard Pxpl5 renderng ppelne and analyzng ts latency. Then we wll present the Slats renderng ppelne, pontng out fundamental dfferences from the standard ppelne. Renderng Control Ppelne Descrpton The standard Pxpl5 renderng ppelne s controlled by a software layer called Renderng Control (RC) [ELLS91]. RC assgns one GP the role of master GP (MGP), whle the others serve as slave GPs (SGPs). The MGP synchronzes the host, SGPs, renderers, and frame buffers. The ppelne t controls comprses the followng stages: 1) SEND - Send edtng commands from the host to the SGPs 2) EDIT - Execute edtng commands 3) TRANSFORM - Transform prmtves 4) RENDER - Render prmtves 5) COPY - Copy pxels to the frame buffer See fgure 1 for an example of ths ppelne. Durng the SEND stage, an applcaton runnng on the host makes some calls to set attrbutes (such as color), create and edt prmtves (such as trangles or spheres), and set transformatons, ncludng the vewng transformaton. These commands are stored n a dsplay lst, whch the host dstrbutes to the approprate SGPs. The host typcally dstrbutes prmtves among the SGPs n a round-robn fashon to promote even load balancng and broadcasts attrbutes and transformatons to all the SGPs. Typcal applcatons send few edtng commands from frame to frame because of the lmted communcatons bandwdth out of the host workstaton. At some pont, the applcaton decdes that t s through sendng edtng commands and requests that the MGP begn the new frame. Ths request marks the end of the SEND stage. The host may not begn the SEND stage for the next frame untl the EDIT stage from the current frame fnshes.

9 9 Before the EDIT stage can begn, several condtons must be met: 1) The SEND stage from the current frame s fnshed. 2) The TRANSFORM stage from the prevous frame s fnshed. 3) The RENDER stage from the 2nd prevous frame s fnshed. 4) The COPY stage from the 3rd prevous frame s fnshed. When the MGP learns that the above condtons are met, t ntates the EDIT stage by tellng the SGPs to begn applyng to ther local dsplay lsts the edtng commands they have receved from the host. For many typcal applcatons, ths stage s bref because dsplay lsts reman largely unchanged from frame to frame. Each SGP can start ts TRANSFORM stage as soon as t fnshes applyng ts edtng commands. The 2nd prevous RENDER stage must also be fnshed, but ths condton s guaranteed to be met before the EDIT stage can begn. So we can see that the start of the TRANSFORM stage requres no synchronzaton by the MGP. Durng the TRANSFORM stage, each prmtve s frst transformed and then bntzed. The SGP apples some lnear transformatons to each prmtve to express the prmtve n terms of a screen coordnate system rather than an object coordnate system. It uses the screen coordnates to determne whch screen regons each prmtve may le n and to generate renderer commands. As mentoned n the secton about the renderers, these commands typcally nclude edge equatons and plane equatons for addng prmtves to a Z-buffer. These commands are stored n bns, each of whch holds commands for one 128x128 screen regon. Each SGP actually mantans two bn contexts, each of whch has one bn per screen regon. The TRANSFORM stage uses one context for bntzng the current frame and the other context for storng the prevous frame s commands untl they are sent to the renderers. For prmtves fallng nto multple regons, the SGP places commands (or, rather, ponters to these commands) n every applcable bn of the current frame s bn context. When all the prmtves have been transformed and ther commands have been generated and bntzed, the SGP announces the completon of ts TRANSFORM stage to the MGP. When the MGP learns that all the SGPs have fnshed ther TRANSFORM stages, t ntates the RENDER stage by assgnng a screen regon to every renderer and dstrbutng renderer tokens. The MGP gves hghest prorty to the screen regons wth the most commands watng n SGP bns by assgnng these regons frst. The MGP then sends the token for each assgned regon to a dfferent SGP, whch receves the token, sends out the bn for the token s regon to the token s renderer, and forwards the token to the next SGP on the rng. The next SGP follows the same procedure. The last SGP to receve the token sends not only ts

10 10 bntzed commands, but also the end of frame commands for that regon, whch typcally nclude some deferred shadng commands. Ths SGP then returns the token to the MGP, whch assgns the assocated renderer to the next unrendered regon. If there are no more unrendered regons, the RENDER stage for ths frame s complete. The COPY stage s loosely synchronzed wth the RENDER stage. Copyng a regon to the frame buffer does not requre that the entre RENDER stage be completed. As long as a renderer has fnshed renderng a regon and the frame buffer s avalable, the renderer may copy the regon to the frame buffer. The frame buffer s consdered avalable when half of the double buffer s beng scanned out to the dsplay, whle the other half contans data that were scanned out earler. The renderers can safely wrte to ths old half. If the buffer s avalable when the MGP frst sends out a renderng token, t sets a flag allowng the last SGP to nclude the copy to frame buffer command n ts end of frame commands. If not, the MGP s responsble for ssung the command for that regon. For an nterlaced stereo confguraton, Pxpl5 uses two nterlaced frame buffers (one for each eye). Renderng Control renders the stereo par by puttng frst the transformaton for the left eye, then the transformaton for the rght eye, nto the ppelne. It consders the par as one frame, so all edtng commands are performed before renderng the left screen, and only the vewng transformaton s changed before renderng the rght screen. The SGPs and renderers know whch vewng transformaton, bn context, and frame buffer to use for each eye s pxel data. See fgure 2 for an example of ths dual frame buffer stereo ppelne. For dscusson of ths confguraton, we need to modfy our descrptons of the condtons to start each ppelne stage. To begn the next SEND stage on the host, the EDIT stage from the prevous frame must be fnshed, just as n a sngle frame buffer confguraton. To begn the next EDIT stage, the followng condtons must be met: 1) The SEND stage from the current frame s fnshed. 2) The TRANSFORM stage from the prevous frame s fnshed. 3) The RENDER stage from the 2nd prevous eye s fnshed. 4) The COPY stage from the 3rd prevous eye s fnshed. To begn the next TRANSFORM stage, the followng condtons must be met: 1) The EDIT stage from the current frame s fnshed. 2) The RENDER stage from the 2nd prevous eye s fnshed. 3) The COPY stage from the 3rd prevous eye s fnshed.

11 11 Latency analyss Now we are ready to analyze the latency of ths Renderng Control ppelne. For ths analyss we assume that the frst stage, the sendng of edtng commands, s neglgble. We measure latency from the start of the EDIT stage to the end of the COPY stage. The mnmum latency stuaton occurs when the TRANSFORM and RENDER stages are trval (few prmtves). In ths case, the bottleneck s the tme to copy pxel data from the renderers to the frame buffers (see fgure 3). It takes about 780 µs for a renderer to transfer one regon of pxel data to a frame buffer. Each eye dsplay has 20 regons, so the total copy tme s 780 µs/regon * 20 regons/eye * 2 eyes = 31.2 ms. Because each odd or even feld takes 1/60 second (about 16.7 ms), that comes out to almost one feld tme per eye n copy tme alone. Assumng the TRANSFORM and RENDER stages each take half a feld or less, they wll get farther and farther ahead of the COPY stages. Because the ppelne can mantan only a lmted number of bn contexts at each stage, ths results n ppelne bubbles. The transformng and renderng wll each be stalled for half a feld, gvng us a renderng latency of 3 felds (50 ms) from the start of the TRANSFORM stage to the end of the COPY stage. To measure an actual mnmum latency condton, we ran a small applcaton on a small Pxpl5 confgured wth 13 GPs, 5 renderers, and 2 NTSC frame buffers. We rendered a scene wth 12 trangles (1 for each SGP), each coverng the entre screen. Ths forced every GP to communcate wth every renderer, as t would n a typcal scene. The applcaton ran at 30 frames/second. The mean tme from the begnnng of the TRANSFORM stage to the swappng of the double buffer was 56.5 ms (about 3.4 felds). Ths s slghtly longer than we predcted n the precedng paragraph. Ths may be due n part to varyng degrees of synchronzaton between the COPY stage and the vertcal retrace on the frame buffer. It may also reflect the fact that the overhead for communcaton between the GPs and renderers becomes sgnfcant for scenes wth few prmtves. As more prmtves are added to the dsplay lst, the TRANSFORM and RENDER stages take longer, and the latency ncreases. We cannot determne a maxmum latency by analyzng a ppelne dagram. Such an analyss shows us only that the latency can ncrease wthout bound. The maxmum latency n practce occurs when the applcaton reaches a lmt to the length of the ppelne stages t can handle, whch s determned by the maxmum number of prmtves that can be stored n each GP s dsplay lst and bns. Addng transformatons as well as prmtves to the dsplay lst can also ncrease the length of the TRANSFORM stage, but we are currently consderng only data sets n whch prmtves vastly outnumber transformatons n the dsplay lst. We measured the maxmum latency usng the same applcaton as we used to measure the mnmum latency, but we dramatcally ncreased the number of

12 12 trangles. Our applcaton could send up to 7700 full-screen trangles wthout overloadng the bn memory on the GPs. Ths number does not reflect the maxmum number n a typcal applcaton, however. Although only 7700 trangles are transformed, full-screen trangles are a pathologcal case for the RENDER stage. Trangles n typcal applcatons seem to affect only 1.3 regons (per eye) on the average, but these trangles affect 20 regons each, usng over 15 tmes the bn memory used by more typcal trangles. The applcaton ran at 2 frames/second wth a mean frame tme of 293 ms, or about 17.5 feld tmes. Of course, the latency s less of an ssue at ths pont than the unacceptable frame rate. Nether of these analyses s really approprate for typcal applcatons. In fact, the ppelne dagram for a typcal HMD applcaton does not have a steady state. The number of prmtves per regon depends on the vewng transformaton, whch changes from frame to frame. To make some ntermedate analyss of the ppelne, we wll assume that the applcaton tends to use the ppelne hardware effcently (.e., GPs and renderers are always workng). If the TRANSFORM, RENDER, and COPY stages all have equal length and the frame rate s 30 Hz, the latency may be as large as 4 felds (about 67 ms), dependng on how far nto the RENDER stage the COPY stage s begun (see fgure 4). If the TRANSFORM and RENDER stages are of equal length, but longer than the COPY stage, the latency should be 3 tmes the length of the TRANSFORM stage plus from 1 to 2 tmes the length of the COPY stage. To measure the latences of some typcal applcatons, we have nstrumented a fly-through model vewer to make the same latency measurements as our prevously mentoned mn-applcaton. The model vewer allows a user n an HMD to nteract wth a 3D model by walkng and flyng through the model, grabbng the model, and scalng the sze of the model. The followng table summarzes the results: Model Prmtves Mean Latency Std. Devaton Mg 417 trangles 4.61 felds felds Molecules 2,620 trangles, 4.47 felds felds 1,219 spheres head 59,592 trangles 8.13 felds 2.48 felds The renderng latency for these nteracton sequences vares not only wth the number of prmtves n the scene, but also wth the arrangement of the prmtves and the path of the user s head. The mean latency and the standard devaton are coarse measures that can change a great deal from sesson to sesson.

13 13 Slats Renderng Ppelne Descrpton Slats s a sngle feld-tme latency renderng system that runs on Pxpl5 n place of the standard Renderng Control system. The current prototype uses 1 GP, 5 renderers, and 2 NTSC frame buffers. The goal of ths system s to serve as an example of low latency renderng and to nvestgate the benefts of renderng wth only 1 feld tme of latency. We dd not optmze the system for the greatest possble throughput but nstead focused on achevng the proposed latency goal and ensurng the correctness of the dsplayed mage. Further optmzaton mght mprove throughput dramatcally, but would also complcate the correctness problem by forcng us to trade latency for correctness n certan worst cases. The Slats ppelne comprses the followng stages: 1) SEND - Send vewng transformatons for both eyes from host to GP 2) TRANSFORM - Transform and bntze trangles 3) RENDER - Render trangles 4) COPY - Copy pxels to the frame buffer See fgure 5 for an example of ths ppelne The SEND stage s a smplfed verson of RC s SEND stage. Because ths s a prototype system, the host sends the prmtves only once durng ntalzaton and cannot edt them durng executon. So the only data the host sends through the ppelne are the two vewng transformatons convertng from world coordnates to left and rght screen coordnates. Ths smplfcaton has lttle effect on our comparson wth RC because we assumed that few edtng commands were sent under the RC system anyway. Unlke Renderng Control, the SEND stage never requests that a new frame be drawn; the host just acqures trackng data, calculates the vewng transformatons, and sends these transformatons to the GP as quckly and as often as possble. Though we have placed ths SEND stage n the lst of ppelne stages for comparson wth Renderng Control, t actually does not qualfy as a ppelne stage; t occurs asynchronously wth respect to the true renderng ppelne. For Slats to functon as ntended, new trackng data must arrve at the host 60 tmes a second. If delays occur n recevng trackng data, Slats reuses the most recent trackng data untl new data arrves. Such delays effectvely ncrease the total latency of the system, so they must be nfrequent f we are to observe the system s most responsve behavor. The TRANSFORM stage uses only one GP. Ths mnmzes communcaton between the GPs and contenton for the renderers. A major dfference between Slats s and RC s TRANSFORM stages s that Slats consders both eyes smultaneously, makng only one pass over the prmtves, whereas RC consders them sequentally, makng one pass for each eye (we wll see later how ths accelerates the COPY stage). The GP employs the same transformaton method

14 14 as do the SGPs n RC, but the bntzaton dffers. In the current mplementaton, the GP puts all the renderer commands for drawng trangles nto a sngle bn. Every regon renders ths sngle set of trangle commands. If we assume that a typcal trangle appears n only 1 or 2 regons, we see that renderng the trangle n all 20 regons uses over 10 tmes the necessary bandwdth to send the commands and 10 tmes the necessary renderng tme. Ths waste s justfed for the moment because t ensures that the latency depends only on the number of on-screen trangles and not ther arrangement. Ths makes t easy to determne the maxmum data set sze that Slats s guaranteed to render wthn a sngle feld tme. For ths prototype, that data set sze s 140 trangles. Usng more trangles wll prevent the current Slats prototype from correctly renderng the scene when all the trangles are on-screen. The RENDER stage dffers somewhat from RC s RENDER stage, though both eventually generate pxels for the frame buffers. The most mportant dfference s that the assgnment of renderers to screen regons s statc. Each of the 5 renderers mantans 4 regons -- a top and bottom regon n each of the 2 eyes. (Note that when we consder the odd and even felds separately, each eye has only 10 regons of pxel data, so 5 renderers can render the 20 total regons of a partcular feld.) Ths statc bndng of regons to renderers allows the GP to start sendng renderer commands before t has fnshed transformng the prmtves. In fact, the GP sends commands to all the regons whenever t has enough data to fll a rng message of maxmum sze. The use of maxmum-szed messages mnmzes the message overhead. One consequence of the statc bndng and mantenance of 4 regon contexts per renderer s that the 4 regons must share the 208 bts of pxel memory. Because of ths restrcton, Slats does not perform Phong shadng, whch requres an nterpolated normal at every pxel. It currently uses color nterpolaton on vertex colors precomputed usng a radosty model. These vertex colors could also be computed on GP by evaluatng a lghtng model at each vertex. The COPY stage also s organzed dfferently from the COPY stage of RC. Because RC does each eye sequentally, t can copy to only one frame buffer at a tme. Ths wastes rng bandwdth, because each frame buffer has ts own rng port, makng t feasble to copy to the left and rght frame buffers smultaneously. As soon as the Slats GP s done transformng and sendng trangle commands to the frame buffers, t can send the copy commands. Because the two frame buffers can receve data smultaneously, the 20 regons can be transmtted n the tme t takes to transmt 11 regons sequentally. Here s the reason for the 11 transmsson tmes (rather than 10). If all the renderers send regons alternately to the left frame buffer then the rght frame buffer, there s one copy tme durng whch no data s sent to the rght frame buffer as the transmssons begn and one copy tme durng whch no data s sent to the left frame buffer as the transmssons end. The problem can be elmnated only by havng the GP synchronze every regon copy to ensure a partcular order. The

15 15 tme taken by ths synchronzaton would severely hnder the GP s transformaton capablty. Because the tme requred to transmt all the regons s less than half a feld tme, we can make an optmzaton to the COPY stage. To guarantee proper mage dsplay, the COPY stage n ts orgnal form must fnsh copyng all 20 regons before the occurrence of the vertcal retrace. We can optmze ths because copyng half of the regons takes only one quarter of the feld tme. Ths allows us to copy the top regons before the retrace and the bottom regons mmedately after the retrace (see fgure 6). The GP needs to synchronze these two halves of the COPY stage to ensure that all of the top regons are coped before any of the bottom regons are coped. By allowng Slats to copy only 10 regons before the vertcal retrace, ths optmzaton gves the TRANSFORM and RENDER stages more tme to do ther work and s responsble for rasng the maxmum number of trangles of the prototype system from 110 to 140. Latency Analyss Analyss of ths ppelne n terms of latency s, by desgn, trval. The renderng latency, from the begnnng of transformaton to the begnnng of scanout, s one feld tme, or 16.7 ms. The frame rate s also trval: 60 frames/sec, the same as the NTSC montor. One way to look at the throughput s to say we draw 140 trangles x 2 eyes x 60 Hz = 16,800 trangles per second. Ths measurement stands on rather shaky ground, however, and really pushes the lmts of the defnton of trangles per second. Though we really draw the trangles 60 tmes every second, we draw them at only half the resoluton of the dsplay devce. So the transformaton occurs 60 tmes per second, but full screen rasterzaton really occurs only 30 tmes per second. Problems Encountered Whle we were testng Slats, many problems came up that hndered our testng the effectveness of sngle feld tme renderng for see-through HMD applcatons. These problems fall nto the categores of trackng problems and dsplay problems. If trackng transport tme s not properly managed, t can dwarf the tme savngs we have made by reducng renderng latency. Also, statc regstraton problems, caused by magnetc and optcal dstortons, can cause objects to be ncorrectly postoned n the vrtual world even when the vewer remans perfectly stll. Ths makes the reducton of dynamc regstraton problems, such as latency, harder to detect, because a vrtual object s regstraton wth the real world changes as the vewer moves, even on a system wth zero latency. Trackng Problems We have had 4 trackng systems avalable for research n our department recently: a Polhemus 3-space, a Polhemus Fastrak, an Ascenson Flock of Brds,

16 16 and an optcal celng tracker [WANG90]. Both of the Polhemus trackers have magnetc felds wth easly notceable dstortons, causng a plane of movement to appear bowl-shaped. Our lab envronment, whch has metal n the floors and celng, s far from deal, and contrbutes to ths effect. The Ascenson and optcal celng trackers gave us more relable trackng data, but could not qute match the Fastrak s update rate. The effect of a dstorted magnetc feld s dramatc n a see-through HMD. Because the tracker measures the head s poston and orentaton naccurately, the computer generated objects are not properly regstered wth the real world. Even f we could reduce the system s total latency to zero, the objects would appear to move as the user s head moved. Ths makes t harder to dstngush between object moton caused by latency and object moton caused by poor regstraton. The other trackng problem s not related to the actual trackng hardware, but to the method of transportng the data. We currently get trackng data (except for optcal celng data) by way of a Unx seral port on the host workstaton. Unfortunately, the Unx kernel polls the seral port only at 30 ms ntervals to see whether new data have arrved. Ths adds almost 2 feld tmes to the total latency, whch already ncludes tracker, renderng, and dsplay latences. As a result, trackng data arrve sporadcally on the host, obscurng the effects of lower renderng latency. Ths transport delay affects the standard RC ppelne much less notceably than t does the Slats ppelne, because ths delay s a smaller percentage of the RC ppelne s latency than t s of the Slats ppelne s latency. To remove ths unnecessary latency, we appled a temporary hack to the Unx kernel. Ths hack modfes the kernel to cause an nterrupt on the recept of every character at the seral port. Ths worked wth the Polhemus trackers, but not wth the Ascenson, whch caused the workstaton to crash when run wth the kernel modfcaton. Dsplay Problems The other class of problems relates to the avalable see-through HMDs. Our lab has two types of see-through HMD: vdeo see-through and optcal see-through. The vdeo see-through uses a camera mounted on an HMD to capture mages of the real world. Because we have no depth nformaton about the world, we merge the vdeo mages wth the graphcs by usng color chroma-keyng. The computer renders the mage wth a partcular background color, and all the pxels of that color are later replaced wth pxels from the vdeo. Ths overlayng of computer generated objects onto vdeo mages wthout proper handlng of occluson s one major problem wth our current vdeo see-through. Another problem wth the vdeo see-through HMD s that t s dffcult, f not mpossble, to get the perspectve rght from two camera mages. It mght be

17 17 possble to get a proper perspectve arrangement, but not the one lned up n front of the user s eyes. The vsual result may stll be convncng, though, much lke lookng through a perscope. Because of the dffcult nature of ths problem, we stll use only one camera wth our vdeo see-through HMD. A thrd problem wth the vdeo see-through s dstorton created by the camera optcs. Ths results n statc regstraton problems much lke those from the magnetc feld dstortons. Our other see-through HMD s an optcal see-through system, whch projects the mages through a seres of lenses onto crcular, half-slvered mrrors. Whle wearng ths HMD, you can see the real world as you see t normally. There s no latency nvolved, you can see your hands just where they should be, and you even have your normal perpheral vson. The half-slvered mrrors have the same occluson problem as the vdeo see-through, because we stll have no depth map of the real world. In addton, the mrrors make up only a small porton of your feld of vew, so keepng the computer generated objects where you can see them s dffcult. Even small amounts of total latency can drve the objects past the edges of the optcs, makng t hard to know how bad the problem s. Our current optcal see-through HMD also has regstraton problems. Ths s due partly to lens dstorton but largely to the fact that the head unt s not very sturdy and can bend when subjected to the slghtest pressure. The vdeo see-through has certan advantages. The hardware already exsts n peces, and mountng a camera on an HMD s relatvely easy (calbraton, on the other hand, s hard). Because the user looks only at the dsplays n the HMD, the computer can control all the user s vsual nput. Also, much lke the non-seethrough HMDs, the use of vdeo nput nstead of drect sensory nput detaches us slghtly from the real world, makng the computer generated addtons easer to accept. On the other hand, ths does not advance the cause of makng them seem truly real. The optcal see-through s currently more lmted than the vdeo see-through n ts ablty to put objects anywhere n your feld of vew, but t allows you to look at and touch the real world n the most natural way. Future Work Slats stll has plenty of room for mprovement. We can enable Slats to handle larger data sets by performng the bntzaton step correctly, sendng trangles only to the necessary regons. Usng larger data sets wth such a system may cause t to run longer than one feld tme for some vewponts. We thnk we can use the avalable frame buffer bandwdth to copy the most recent complete feld nto a backup feld n case we mss a vertcal retrace. Of course, for the system to

18 18 render wth a sngle feld tme of latency, such msses must be rare. For ths more complex system, the acceptablty of a data set wll depend not only on the number of trangles t contans, but also ther arrangement. We mght also ncrease the throughput of Slats by addng multple GPs. Ths would add a problem of contenton for access to the renderers, but we could presumably ncrease our throughput substantally, because the renderers can handle many more prmtves than we can generate wth such a short TRANSFORM stage, even f we use many GPs. Most mportantly, we need to come up wth new approaches to achevng low latency. We strongly encourage other researchers n ndustry and academa to nvestgate ways of tacklng the latency problem. These wll nclude new ways of usng our current hardware as well as ways to desgn future hardware wth latency n mnd. Low latency renderng, when coupled wth predctve trackng, should help us to acheve zero latency systems n the future and to create more natural vrtual envronments.

19 19 References BAJU92 BISH93 ELLS91 FEIN93 FUCH89 MINE93a MINE93b REGA93 TEBB92 Bajura, M., H. Fuchs, R. Ohbuch, Mergng Vrtual Objects wth the Real World: Seeng Ultrasound Imagery wthn the Patent, SIGGRAPH 92, Vol. 26, No. 2, pp Bshop, G., No Swmmng, Unversty of North Carolna Department of Computer Scence, HMD Research Project Goals. Ellsworth, D., Pxel-Planes 5 Renderng Control, Unversty of North Carolna Department of Computer Scence Software Documentaton. Fener, S., B. MacIntyre, and D. Selgmann, Knowledge-based Augmented Realty, Communcatons of the ACM, Vol. 36, No. 7, pp Fuchs, H., J. Poulton, J. Eyles, T. Greer, J. Goldfeather, D. Ellsworth, S. Molnar, G. Turk, B. Tebbs, and L. Israel, Pxel-Planes 5: A Heterogeneous Multprocessor Graphcs System Usng Processor- Enhanced Memores, SIGGRAPH 89, Vol. 23, No. 3, pp Mne, M., Characterzaton of End-to-End Delays n Head- Mounted Dsplay Systems, Unversty of North Carolna Department of Computer Scence Techncal Report TR Mne, M., and G. Bshop, Just-In-Tme Pxels, Unversty of North Carolna Department of Computer Scence Techncal Report TR Regan, M., and R. Pose, An Interactve Graphcs Dsplay Archtecture, Department of Computer Scence, Monash Unversty, Clayton, Vctora, Australa. Tebbs, B., U. Neumann, J. Eyles, G. Turk, and D. Ellsworth, Parallel Archtectures and Algorthms for Real-Tme Synthess of Hgh Qualty Images usng Deferred Shadng, Unversty of North Carolna Department of Computer Scence Techncal Report TR WANG90 Wang, J. F., R. Azuma, G. Bshop, V. Ch, J. Eyles, and H. Fuchs, A Demonstrated Optcal Tracker wth Scalable Work Area for Head- Mounted Dsplay Systems, Proceedngs of 1992 Symposum on Interactve 3D Graphcs, March 1992, pp

20 20 Fgures SEND SEND SEND EDIT TRANSFORM EDIT TRANSFORM EDIT TRANSFORM RENDER RENDER RENDER COPY COPY COPY Fgure 1: Renderng Control ppelne for frames, +1, and +2

21 SEND SEND SEND E TL TR E TL TR RL RR RL RR CL CR CL CR Fgure 2: Renderng Control ppelne usng dual frame buffer stereo E=EDIT, TL=TRANSFORM left, TR=TRANSFORM rght, RL=RENDER left, RR=RENDER rght, CL=COPY left, CR=COPY rght

22 TL TR TL TR TL RL RR RL RR RL CL CR CL CR Fgure 3: Renderng Control ppelne usng dual frame buffer stereo for a mnmum latency applcaton TL=TRANSFORM left, TR=TRANSFORM rght, RL=RENDER left, RR=RENDER rght, CL=COPY left, CR=COPY rght

23 TL TR TL TR RL RR RL RR CL CR CL +1 CR +1 Fgure 4: Renderng Control ppelne usng dual frame buffer stereo wth mnmum latency for 100% utlzaton of hardware TL=TRANSFORM left, TR=TRANSFORM rght, RL=RENDER left, RR=RENDER rght, CL=COPY left, CR=COPY rght

24 24 TRANSFORM TRANSFORM TRANSFORM RENDER RENDER RENDER COPY COPY COPY Fgure 5: Slats ppelne wthout COPY optmzaton TRANSFORM TRANSFORM TRANSFORM RENDER RENDER RENDER COPY Top COPY Bottom COPY Top COPY Bottom COPY Top COPY Bottom Fgure 6: Slats ppelne wth COPY optmzaton