Bringin. Andrew. Cambridge, CB3. once the. sensing multiple touch. the entire hand. of fine-grained. to approach hides.

Transcription

1 Bringin ng Phyi to the Surfae Andrew D. Wilon 1, Shahram Izadi 2, Otmar Hillige 2, Armando Garia-Mendoza 2, David Kirk 2 1 Mirooft Reearh 2 Mirooft Reearh Cambridge One Mirooft Way 7 JJ Thomon Avenue Redmond, WA Cambridge, CB3 0FB {awilon, hahrami}@mirooft.om, otmar.hillige@ifi.lmu.de, {armandog, dakirk}@mirooft.om ABSTRACT Thi paper explore the interetion of emerging urfae tehnologie, apable of ening multiple ontat and of- gine. We define a tehniquee for modeling the data ened from uh urfae a input within a phyi imulation. Thi afford the uer the ability to interat with digital ob- jet in way analogou to manipulation of real objet. Our tehnique i apable of modeling both multiple ontat point and more ophitiated hape information, uh a ten hape information, and advaned game phyi en- the entire hand or other phyial objet, and of mapping thi uer input to ontat fore due to frition and olli- ion within the phyi imulation. Thi enable a variety of fine-grained and aual interation, upporting finger- baed, whole-hand, and tangible input. We demontrate how our tehnique an be ued to add real-world dynami to interative urfae uh a a viion-baed Our approah hide from appliation developer many of the omplexitie inhe- rent in uing phyi engine, allowing the reation of ap- pliation without preprogrammed interation behavior or geture reognition. ACM Claifiation: H5.2 [Information interfae and preentation]: Uer Interfae. - Graphial uer interfae. General term: Deign, Human Fator, Algorithm Keyword: Interative urfae, game phyi engine tabletop, reating a fluid and natural experiene. INTRODUCTION Emerging interative urfae tehnologie allow uer to interat with the digital world by diretly touhing and ma- nipulating onreen ontent. People who ue uh ytem often omment that thi ability to touh digital ontent add a phyial or tangible quality to the interation, making the virtual feel more real. Many interative urfae-baed ap- pliation attempt to further highlight thi peudo- phyiality by arefully deigning interfae objet that exhibit a ene of real-world behavior. One example i the ommon rotate and tranlate behavior found in many table- ing a heet of paper on a flat urfae with one or more fin- ger [17, 18]. Although uh an interation may feel reali- ti, it i very muh preprogrammed. For example, the way Permiion to make digital or hard opie of all or part of thi work for top appliation, where the interation i analogou to mov- peronal or laroom ue i granted without fee provided that opie are not made or ditributed for profit or ommerial advantage and that opie bear thi notie and the full itation on the firt page. To opy otherwie, or republih, to pot on erver or to reditribute to lit, require prior peifi permiion and/or a fee. UIST 08, Otober 19 22, 2008, Monterey, California, USA. Copyright 2008 ACM /08/10...$5.00. in whih a digital photo rotate or tranlate i defined by peifi interation logi. Suh tehnique thu poe an inherent ripted nature whih may break down one the uer interat with the ytem in way unantiipated by the developer. Figure 1. Some example of phyi-enabled inte- and ration upported by our tehnique. Gathering piling objet (top left), interating with a ball uing olliion and frition (top right) ), folding a loth-like meh (bottom left, middle), and tearing a meh (botupporting tom right). In thi paper we take a different approah for peudo-phyial interation on urfae tehnologie ytem that are often apable of ening multiple touh point and poibly even more ophitiated hape informa- in om- tion. Speifially, we utilize phyi engine ued puter game that imulate Newtonian phyi, thu enabl- ing interation with digital objet by modeling quantitie uh a fore, ma, veloity, and frition. Suh engine allow the uer to ontrol and manipulate virtual objet through et of parameter more analogou to thoe of realworld interation. While phyi engine are omprehenive, they are alo omplex to mater. Many oeffiient and parameter are expoed to appliation developer, and ontrolling the i- when mulation via uer input i non-trivial, partiularly onidering more than a ingle ontat point. We preent a imple yet powerful tehnique for modeling rih data, ened from urfae tehnologie, a input within the phy- ontat i imulation. Our approah model one or more point, uh a thoe ened from a multi-touh urfae [7,14],, and alo ale to repreent more ophitiated hape uh a outline of the whole hand or tangible obto interat jet on the urfae [20,25]. Thi allow the uer with objet by frition fore and olliion, but avoid 67

2 expoing uer and appliation programmer to the omplexitie of the phyi engine. We demontrate the appliability of the tehnique uing a ommerially available game phyi engine and a prototype viion-baed interative urfae. We highlight ome of the interation uh an approah afford, for example gathering multiple digital objet or fine ontrol of a virtual ball uing frition and olliion fore, a hown in Figure 1. Our ytem reate natural and fluid phyi-baed interation for free i.e., without the need either to expliitly program thi behavior into the ytem or to reognize geture. We alo demontrate the ability of advaned phyi imulator to enable uer interation with more omplex material uh a oft bodie and loth, a hown in Figure 1. We have experimented with variou alternative for imulating urfae input within the phyi world, the trade-off of whih are diued in thi paper. Our aim i to allow pratitioner to undertand the nuane of thee alternative o that they may further explore the interetion between interative urfae and phyi. We alo diu both early experiene uing our tehnique and ome of it limitation. RELATED WORK Muh work ha reently been publihed on interative urfae, and partiularly on diret-input tabletop [7,14,25]. Some of thi work examine phyi-like interation. For example, Kruger et al. [17] explored imple notion of frition and motion to allow an objet to be tranlated and rotated uing a ingle point of ontat. Other ytem [12,22] have implemented geture uh a fliking, throwing, and puhing to add veloity and inertia to onreen objet. Beyond tabletop, a number of interative ytem have explored the phyiality of objet a a priniple around whih interation i organized. Eletroni file tak like real doument [19], move out of the way a if they have a olid form [23], peel bak like paper [8], and move a if alive [5]. Rather than employing a full phyi model, uh ytem ue peudo-phyi in minimal way to upport ubtle interation poibilitie. The preent work look to addre a riher et of phyi-enabled behavior, and foue on problem related to input. Variou attempt to provide riher and more realiti 3D interation have alo been explored in the ontext of interative tabletop [13]. Stahl et al. [24] deribe a tabletop where objet float to the urfae when aeed and ink bak to the ground when no longer ued. Hanok et al. preent a et of method to ompenate for off-axi viewing [16] on multi-uer tabletop, a well a tehnique for hallow-depth interation [15]. Thee ytem allow bai manipulation of 3D objet but do not model interation baed on a phyi imulation. Realiti dynami imulation ha a long hitory in graphi and animation ommunitie. Baraff [3] give an overview of the main onept for alulation of rigid-body dynami. More reently, real-time olliion detetion and repone with table frition alulation beame feaible [4]. Erleben et al. [9] provide an overview of urrent tehnique and advanement that led to the development of ophitiated phyi engine for imulation and gaming, uh a PhyX, Havok, Newton, and ODE. However, thee tehnique have yet to reah the uer interfae and interation reearh ommunitie broadly. One notable exeption i BumpTop [1], whih ue a phyi engine to add real-world dynami to the Tablet PC dektop. It upport notion uh a olliion, ma, and inertia, and higher level ontrut uh a piling. The work niely demontrate ome of the apabilitie of a modern phyi engine. However, the approah i baed on a ingle point of input and menu-baed eletion. Compared to the rih mean available for manipulating phyial objet, thi ingle-point input model an be limiting. Multiple input are onidered in the appliation of an early phyi engine by Baraff with the Reponive Workbenh [10]. Thi ytem imulate a 3D bimanual aembly tak uing two 6 DOF input devie and a tereoopi diplay. An objet may be manipulated with one hand by plaing eight pring onneted to the orner of a virtual ube rigidly attahed to the uer hand. Bimanual interation are upported by the uperpoition of fore from both hand. Thu, when one hand i releaed, the objet nap to the poition and orientation of the other. Interative urfae, partiularly viion-baed ytem [14,20,25,26], allow the apture of rih enor data that not only inlude multi-touh data but alo detailed hape information, uh a image of the uer entire hand or other tangible objet near the urfae. We preent a tehnique for modeling thi rih and divere enor input, repreenting thee effetively a frition and ontat fore in the phyi imulation. Thee apabilitie augment and extend the ingle ontat model utilized in BumpTop and the bimanual approah in Reponive Workbenh, upporting ingle-touh, multi-touh, ontour-baed, and tangible input uing a ingle tehnique. INTERACTIVE SURFACE INPUT A ontat on an interative urfae (e.g., a fingertip touhing the urfae) i mot eaily repreented a a direte 2D point. In the ae of viion-baed interative urfae, neighboring enor pixel are uually grouped into ontiguou region or onneted omponent [6], with the idea that eah omponent orrepond to a ontat. The enter of the omponent i then eaily alulated. Thi approah thu redue eah ontat to a point-baed repreentation, regardle of hape. Thi point repreentation of ontat allow appliation developer to think in term of familiar point-retangle hitteting algorithm typial of traditional uror-baed ytem, but it impoe ignifiant limit on interation. Firt, point-baed hit teting may fail to ath the uer touhing a virtual objet if the ontat i not ompat, a when the uer plae a large part of their hand on urfae. In thi ae, the enter point may lie outide the target objet. Seondly, traking point-baed ontat to dedue motion an lead to diffiult problem related to orrepondene. For example, onider two finger that move o near to eah other that 68

3 they now appear a a ingle ontat. The hoie of whih of the original ontat to eliminate an reult in very different motion interpretation. Finally, reduing ontat to point prevent uer from drawing on the full petrum of manipulation tyle found in everyday life. Conider the multiple graping trategie illutrated in Figure 2, for example. Eah give a different feeling of ontrol in the real world. Ultimately, it eem that point-baed ytem enourage the exluive ue of index finger on interative urfae. Figure 2. Multiple graping trategie for rotating a reting objet. One approah for preerving more information about ontat hape i to determine the bounding box of the ontat, or the major and minor axe of an ellipe model that approximately fit the hape. Thee approahe work well for ompat ontat (e.g., fingertip) and ertain hand poe [27], but le o for omplex hape and their motion. Alternatively, the hape may be repreented more preiely a a polygon meh by alulating eah ontat ontour, repreented a a loed path [11]. Another tehnique i to take pixel lying near the ontour by omputing the patial gradient uing a Sobel filter [11]. Thee approahe allow u to upport even more ophitiated repreentation of uer input. We would like to ombine thi broad fidelity of input with advaned phyi imulation to expand the voabulary with whih we an manipulate digital objet. Our aim i to make manipulation of digital objet le ripted, uing rih and varied interation tehnique and trategie. PHYSICS SIMULATIONS Today phyi engine enable the reation of real-world mehani and behavior in graphial appliation while hiding omputational omplexity. They employ many phyi onept uh a aeleration, momentum, fore, frition, and olliion. In addition to rigid bodie, many ytem model partile (for moke, dut, and o forth), fluid, hair, and lothe. Virtual joint and pring give rag doll harater and vehile appropriate artiulation, and material an be programmed with peifi propertie o that ie i lik, for example. The preent work primarily onern ontat fore, uh a thoe due to olliion and frition between imulated bodie. The handling of olliion i typially divided into olliion detetion, the determination of whether two rigid bodie are in ontat, and olliion repone, the appliation of appropriate fore if they are in ontat. For example, the olliion of a ube falling on the floor may be deteted by onidering the interetion of the fae defining the ube with thoe of the floor. The hange in motion of the ube a a reult (the repone) i a funtion of ma, inertia, veloity, the point of ontat with the floor, and other fator. Frition fore reit motion when the urfae of one body tay in ontat with the urfae of another. If two urfae are moving with repet to eah other, kineti frition oppoe the fore moving the bodie. If two urfae are at ret relative to eah other, tati frition oppoe fore that would otherwie lead to the motion of one of the bodie. SURFACE INPUT WITHIN A PHYSICS SIMULATION In order to interat appropriately with virtual objet in a phyi engine, urfae ontat mut be repreented within the imulation. Thee engine have enormou potential and flexibility. Aordingly, there are many trategie for modeling urfae input in the phyi world. We briefly deribe thee trategie here, and give more detail later. Diret fore: A fore i applied where a ontat point touhe a virtual objet. The fore diretion and magnitude i alulated from the ontat veloity and ize if available. Virtual joint and pring: Eah ontat i onneted to the virtual objet it touhe by a rigid link or pring, o that the objet i dragged along with the ontat. Proxy objet: Contat point are repreented a rigid bodie uh a ube or phere. Thee bodie are an approximation of the ontat, and interat with other virtual objet by olliion and frition fore. Partile: Where additional information about a ontat hape i available, multiple rigid bodie or partile are ombined to approximate the hape and motion of the ontat more aurately. Thi allow for better modeling of interation with the whole hand or other ontat uh a tangible objet. Deformable 2D/3D meh: Another approah for modeling more ophitiated hape i to ontrut 2D or even 3D mehe if appropriate enor are available. It would eem that a deformable 3D meh of the hand would ahieve the highet degree of fidelity. But a number of diffiultie exit with thi approah. Firt, mot interative urfae provide ening at or near the urfae only, not full 3D hape. Similarly, beaue the manipulated objet exit only on the (flat) diplay urfae, the 3D hape of the hand, if aptured, would not onform to the objet and o would not reflet the hape of a real hand graping a real objet. Finally, ontruting uh an animated meh i diffiult, requiring robut traking of feature and aurate deformation of the 3D objet. That leave u with a key quetion that motivate thi paper: How doe one bet ue urfae input to interat with advaned phyi imulation in ueful way? We deribe our rationale and experiene in implementing and evaluating the above alternative. The main ontribution of thi paper i a novel Partile Proxy tehnique that retain mot of the benefit of meh-baed repreentation in partiular, a high fidelity of interation but i oniderably eaier for appliation programmer to implement. Applying External Fore Diretly A typial trategy for moving an objet on an interative urfae in repone to touh i to ontinually update it 69

4 poition to math the touhing ontat poition. Generally we will refer to thi manner of moving objet by etting it poition and orientation diretly generally a kinemati ontrol. Within a phyi imulation, however, the mot ommon way for an appliation to ontrol the movement of a rigid body i to apply one or more fore. For example, a paehip in a game might have thruter on either ide of it body. The hip may be propelled forward by applying forward fore at the loation of both thruter. If one of the fore i applied in the oppoite diretion, the hip will turn. Rotation i the by-produt of torque, whih our when fore are applied off-enter (of ma) beaue different part of the body are moving at different peed. From a programmer point of view, thi approah i very different than moving the hip by etting it poition. To effet kinemati ontrol within a phyi imulation, we mut alulate the preie fore and torque required to move the objet into it target poition. Thi method of poitioning an objet enure orret olliion repone with other bodie in the imulation. In omparion, diretly etting the poition of the body within a imulation an lead to untable and unpreditable reult. Abolute poitioning might be analogou to teleporting a real objet from one loation to another. Iue uh a interpenetration whereby objet beome partially embedded in eah other, an arie. A natural trategy for moving an objet to follow a ontat on an interative urfae i therefore to onider that eah ontat impart a frition fore to the body it touhe aording to the ontat motion (and preumed ma). Thee multiple frition fore may be applied to the body, a in the example of the paehip. Unfortunately, to alulate the fore neeary to math a ontat movement, all other external fore ating on the body mut be taken into aount and ounterated. Thee may inlude frition fore and olliion repone that are diffiult or impoible for appliation developer to obtain. Thi diffiulty extend to onidering fore orreponding to urfae ontat. In the ae of multiple ontat, the orret frition fore orreponding to eah ontat mut be determined imultaneouly. Conider the ae where one or more of the ontat exhibit tati frition. Reall that tati frition exert a fore that ounterat fore that would otherwie lead to a body motion. For example, if one ontat pin an objet o that it will rotate due to the motion of another ontat (e.g, Figure 2, left), the appliation of orret frition fore due to one of the ontat require knowing the frition fore due to the other. In fat, at the heart of any phyi engine i a ophitiated ontraint olver that addree thi very problem. Without eentially ontruting a new olver within the phyiengine, or without ae to internal of the exiting olver, it would eem impoible to orretly apply ontat fore diretly. Even if it were poible to hange the olver or embed another, uh an approah would go againt the pirit of the preent work, wherein an exiting full-featured phyi engine i leveraged rather than built from rath. One poible olution i to treat all frition a kineti. But thi poe a problem in the pinning example. Beaue kinemati frition fore only at in the preene of relative motion, the ounterating fore that keep the pinned part of the objet tationary mut firt move. Thu, thi approah reult in a omewhat viou and lightly unpreditable feel when moving objet. Conneting to Objet with Joint and Spring Another kinemati approah, ued in ytem uh a BumpTop [1], i to onnet virtual objet and an input ontat uing a joint. Think of thi a an inviible piee of rope of predefined length that i tied to the objet at a partiular anhor point. The objet i then pulled along uing thi rope. By attahing a joint off-enter, the objet i ubjet to both fore and torque allowing the objet to move and rotate uing a ingle onnetion. In our earlier pinning example, one joint attahing a tationary ontat point to one orner of the objet would erve a a pivot point. A eond joint attahing a eond moving ontat point to an oppoing orner would aue the objet to pin around the firt ontat point. Thi approah i not well uited for multiple imultaneou ontat point, partiularly thoe pulling in oppoite diretion. Wherea in the real world, multiple ontat pulling in oppoite diretion on an objet would reult in the finger liding, or the objet deforming or tearing, neither behavior i upported by joint ontraint on a rigid body. It i thu eay for multiple rigid ontraint to overontrain the imulation, reulting in numerial intability and unpreditable behavior. Spring an in part alleviate ome of thee iue by providing more flex in the onnetion. However, a trade-off exit between the elatiity of the pring and how reponive the onneted objet i to ontat motion (pring hould be fairly hort and rigid to allow for a fater repone). Problem of numerial tability and unontrolled oillation are likely [10]. Another approah i to allow the joint or pring to break in thee ituation, but thi an eaily lead to ituation where objet fly out of the uer reah. SETTING THE SCENE FOR A NEW TECHNIQUE We have o far deribed two tehnique that one would typially employ in ingle-point phyi-enabled appliation, and diued the limitation of both in term of modeling multiple ontat. The modeling of uh input i hallenging but only part of the tory with repet to the limitation of thee approahe. Firt, a we deribed earlier, ontat are not alway direte 2D point, and it may be deirable to math the hape of the ontat input loely. It i unlear how one would model more ophitiated hape and ontour with either of thee initial approahe. Seond, the above tehnique addre the ae where the uer touhe the objet diretly, thereby moving the objet by frition fore. Neither of thee approahe addree the movement of objet by olliion fore, i.e., from ontat fore applied to the ide of the objet (a in Figure 2, middle). 70

5 The next etioon preent a tehnique whiih handle friit tiion and olliioon fore in th he ame framew work and i eailyy extended to handle hape of arbitrary oontour. In doinng it addreee many of th o, he diffiultie of the previouu tehnique. g objet byy touhing fore to puh objet around or grab f two oppoing ide. them from A maall hange in thhe kinemati ontrol enable the proxy objet to exert fritioon fore whenn it lie on top of another rigid body b (a whenn the uer touhe the top uurfae of a virtuall objet, for exxample). In paartiular, only fore tangentiall to the touheed objet are applied a to math the ontat pooition. A wiith regular dynnami bodie, gravity i till innluded a an external e fore. In I the ae whhere gravity i direeted into the urfae, the proxie p thu exxert downward fore onto othher objet annd aue fritiion fore. Thi hybrid h kinematii-dynami onntrol of the objjet an be implem mented by direet addition of fore to the proxy p rigid body, or by a primaati joint ontraint on the body b motion. The T imulated weight w of the finger on the objet o may be adjuuted by hangging the ma of o the proxy obbjet, while the maaterial properttie of the virttual objet may m be adjuted by hanging ttati and kinetii frition oefffiient. P Proxy Objet The idea of prooxy objet i to T t inorporate into the phyi imulation a rigid body for eah urfae ontat. Thee b bodie are kineematially onttrolled to mathh the poition of thhe urfae onntat and an be thought off a inarnationn o ontat poinnt within thee phyi imuulation. Beaue of thhey are regulaar rigid bodie,, they may intterat with othher r rigid bodie in the uual way y: either by olliion or friitiion. The proxy apprroah arrie variou T v benefit uh a hidinng thhe omplexityy of fore alu ulation (in fat, hiding almoot a phyi appet) from thee programmer, while avoidinng all thhe diffiultie of the previo ouly deribedd approahe. It leverage olliion a well a frition fore (both tati annd k kineti) to moddel rih interaation uh a puhing, grabbb bing, pinhing,, and dragging g. Proxy objet interat wiith o other objet inn the imulatio on through the mean provideed b the engine. Finally, thi approah avoid unneearry by train on the olver (e.g., in nerting extrem me fore valuee) a reulting unntable imulattion tate. and The main m advantage of the proxy objet o tehniquue i that it leveragge the built-iin apability of o the phyi engine to imulaate both fritioon and olliion ontat forre. Mot ignifiiantly, beaue the alulattion of ontat fore i handleed entirely by the built-in phhyi engine olver, the ombinned effet of imultaneou tati and kineeti frition fore due to multipple proxy objeet i handledd orretly. Thee frition foree enable uer to tranlate and rotate objet (through opppoing fore) that t they touh diretly. Proxy objet are P a reated and d poitioned for fo eah point of ontat. Mot imply, a ing gle hape prim mitive uh a a ube or phere may be ued for f eah ontaat. When a onntaat initially apppear, a ray aating alulatiion i performeed too determine thhe 3D poition of the proxy o that it touhe thhe underlying objet, a how wn in Figure 3. 3 An interetinng a alternative to uing u a phere or ube a a proxy p hape i to reate a thin aapule, box, or ylinder whih trethe from thhe 3D amera near plane to the urfae of the touhed obbjet (ee Figuree 4). Thi kind d of proxy will ollide not onnly w objet reting on the am with me plane a thee touhed objeet (or floor ), buut alo objet that are in midd-air, or takeed o other objet. Suh beh on havior may oorrepond more loely to uer expetation. From Point to Con ntour: Partile e Proxie Thu far f we have appproximated eah touh point a a ingle proxy objet. Thi permit p a impple, fat impleementation, and lennd itelf to eening ytem that report onnly ontat poitioon and no happe informationn, a well a apppliation that faavor interationn with the fingeertip or tylu. Some interative urrfae provide hape informaation, uh a an oriented o ellipee, bounding boox, or full polyygonal ontour. The T idea of thee partile proxy xy i to model the t ontat hape with a multitudde of proxy obbjet ( partilee ) plaed along the t ontour of the ontat (eee Figure 4). Partile are added and removed a ontour hhange ize andd hape. A pratial implementtation involve reating a new n et of proxy objet for thee ontour at thee beginning of eah imulation frame, and detroying d all proxy objet after the phyi imulation ha been upddated. Even though t the proxiee will be detroyed after thhe phyi uppdate, eah enat olliion and frition f fore during the upddate. Figure 3. Proxy objet (rred) are plaed d in the 3D ene, one per ontat (lefft image), by ra ay ating. A the eningg ytem provid A de update too a ontat poitiion, the orreponding proxy y objet i kinnematially onntrrolled to math the updated poition. Thi i done, a ded ribed earlier,, by applying the neeary fore to brinng thhe proxy objet (of known ma) m to the upddated poition of thhe ontat. Thhi heme allo ow uer to leeverage olliioon Fig gure 4. Partile e proxie appro oximate the ha ape of mu ultiple ontat. Left: applyin ng frition from m the top p, and olliion from the ide to grip a blokk. Middle e: Long proxy partile objet (red) illuttrated. Rig ght: Partile proxie aomm modate non-fla at objet. Here we mo odel eah proxxy a a long ap pule. 71

6 The advantage of the partile proxy approah i twofold. Firt, olliion appear to be more orret beaue they more loely follow the hape of the ontat. Thi i parti- ularly important when uing the flat or ide of the hand, tangible objet, or generally any ontat other than fin- gertip (ee Figure 5). Similarly, the ditribution and mag- nitude of frition fore on the top of an objet are more aurately modeled. For example, the flat of the hand may exert more frition than the tip of a finger (Figure 2, right) by virtue of having more partile aigned to it. Likewie, a ingle ontat turning in plae an exert frition fore to rotate an objet. Unlike the ingle proxy model, eah par- onform to irregularly-haped 3D virtual objet (Figure 4). A in the ingle proxy objet model, eah partile i kine- matially ontrolled to math the movement of the ontat to whih it belong. Generally, the veloity of a point on tile i plaed (ray at) eparately, o that a ontat an the ontour an be omputed by examining the ontat ontour in the previou frame. Thi alulation may be imple, a with an ellipe model, or more omplex, a with a polygonal ontour. altogether by plaing proxy partile at image loation with high patial gradient (e.g., Sobel filter [11]). Thee pixel will lie on ontat ontour. The partile proxy teh- nique i ummarized a: ompute Sobel image from urfae input for eah pixel with high patial gradient: ray at into ene to determinee initial partile poition add partile rigid body to phyi imulation ompute ontat motion at partile (e.g., from flow) ompute orreponding tangential motion in ene apply fore to partile to math ene motion apply downwardd fore (gravity) to partile update phyi imulation detroy all partile rigid bodie The intantaneou, pieewie nature of the hape and mo- method tion alulation of the flow-baed partile proxy poee important advantage. Firt, the frition and on- reult tat fore lead to more table phyi imulation than if hape and motion were alulated from direte traked objet. Seond, beaue the tehnique make few aumption regarding the hape or movement of ontat, it impoe few limit on the manipulation a uer may per- or om- form, whether leading to olliion, frition fore, bination thereof. Figure 5. Top left: Partile proxie approximate the hape of multiple ontat and phyial objet (up, notepad). Top right: Sobel image how onthey projet tat ontour. Bottom: Partile (red) a into the 3D ene. From Traking to Flow One diffiulty in baing veloity alulation on traked ontat i that t traking an fail, partiularly when the uer i uing le ontrained graping poture uh a the edge or flat of a hand rather than the more uror-like index fin- ger. In thee ae, omponent an plit and merge in way that do not orrepond to how we ee the phyial input, leading to erroneou veloity alulation, and ultimately in the ae of our phyi imulation to unpreditable motion. An alternative approah i to alulate the motion of the partile independently of any traked ontat information. For example, loal feature of the image may intead be traked from the previou frame to alulate veloity. Sim- ple blok mathing of the ort ued in optial flow [2] i one uh tehnique (ee Figure 6). When uing loal motion etimate, the traking of direte ontat objet and exat ontour may then be avoided Figure 6. Computing flow of a partile. Surfae mo- tion at a point x t i omputed by omparing ue- motion ive edge image. Correponding tangential in the ene i omputed by ray ating image point x t into the 3D ene, to obtain 3D point p t. Point p t+1 i found by projeting image point x t+1 onto the taninput type gent plane formed by p t and objet normal n. NEW PHYSICS-BASED INTERACTIONS Our goal in introduing more detailed urfae into a phyi imulation i to enable a wide variety of ma- While nipulation tyle drawn from real-world experiene. we have only begun to explore ome of the poibilitie that thee tehnique afford, here we onider a few whih we believe are noteworthy. Manipulation Fidelity The ability to exploit detailed hape and motion informa- manipu- tion ha broad onequene when onidering the lation of even the implet objet. Free-moving virtual objet an be moved by any one of a variety of trategie that ombine olliion againt the ontour of hand and finger with tati and kineti frition. Beaue all three kind of fore an be employed imultaneouly, the overall impreion i one of unuually high fidelity. An interetfree to roll ing example i the manipulation of a ball that i on the urfae: it may be ompelled to roll, pin, top, or 72

7 boune in a urpriingly preie fahion, uing a ingle light touh, multiple touhe, or the flat of the hand for topping power (Figure 1). Phyial objet an alo be integrated at no ot, allowing a variety of intereting tang- ible behavior (ee Figure 7 for ome example). Gathering The ability to ene and proe ontour, a well a ditri- bute frition fore pieewie aro the virtual pae, enable the manipulation of many objet at one, muh a one might move a group of mall objet pread aro a table (ee Figure 1, top left). Uer may ue the edge of their hand (or even arm) to ollide againt many objet at one, or ue the flat of multiple hand to apply frition fore. For interative urfae able to ene phyial ob- jet, an intereting poibility i to ue a ruler to move and align multiple objet. Manipulating Objet in 3D Modeling virtual objet and input in 3D enable interet- ing yet familiar interation. For example, a flat objet ret- or applying frition. Depending on the mae and frition involved, it may be neeary to hold the larger objet in plae. It i thu important for the deigner to tune mae, frition, and appearane to math uer expetation. If the interation i limited to olliion fore from the ide and frition fore from the top, however, the manner in whih a uer may plae the maller objet on top of another i unlear. Ramp, ee-aw, and other ontrution are poible, if omewhat ontrived. In ertain ae it may be poible to flip one objet onto another through the applia- tion of uffiient frition fore to one ide of the objet. When the objet to be taked are thin, uh a ard repreenting doument [23, 1], one approah i to give the top and bottom urfae of eah objet a ambered hape that allow the uer to raie one ide by preing down on ing on a larger flat objet may be moved by tapping it ide the other. The uer may then move another like-ized ard under the tilted ard (Figure 7, top left). Thi behavior or- repond to our awarene that in the real world even flat objet uh a ard and paper have ome 3D hape that i often intuitively exploited to manipulate them. Figure 7. A ambered ard i lid on top of another by puhing one ide while holding the bottom ard in plae (top left); tangible objet interat with the digital: e.g., the ide of a phyial piee of ard ued to gather up objet (top middle), or a real up ued to pin down a loth (top right); tearing a meh in two by applying oppoing fore (bottom). Cloth and Soft Bodie We have ued rigid bodie uh a boxe and phere to explain our interation tehnique. However, in the real world many objet are not rigid but are intead oft, malare exerted leable, and an deform or diolve when fore on them. Example inlude rubber, lothe, and paper. In addition to rigid body dynami, mot available phyi imulation offer ome form of upport for oft body, loth, and fluid imulation. A all interation in our model are onduted through olliion or frition fore, the model an be applied to arbitrary virtual objet. For example, it i poible to rumple a piee of loth with a graping intera- loth tion uing all the finger of one hand. The rumpled an then be traightened by puhing down with the flat hand. One an even tear paper-likee objet apart by applying fore in oppoing diretion on two orner (Figure 7). Another poible appliation would allow oft volumetri bodie to be quihed o a to fit into avitie or ompreed o a to lide underneath other objet. Soft mate- rial ould alo be ued for terrain; deformation ould be triggered by uer digging their finger into the terrain, uing their whole hand to form valley, or uing a upping geturee to reate elevation. More open-endedd and freeimulating form interation with partile ytem (e.g., and) and fluid an be imagined in a gaming ontext. USER STUDY To further undertand and evaluate the utility of the tehexperiment nique deribed in thi paper, an exploratory wa performed. The following quetion were addreed: Are uer able to omprehend and exploit the openne of interation that the phyi model afford? I the interation of uffiient fidelity? I it dioverable and preditable? Do uer notie and value the added fidelity, or do they jut expet kinemati ontrol? Ultimately, how do uer expre their expetation in the phyi-enabled manipulation? The tudy expoed 6 partiipant to 3 imple phyi- variou enabled tak (a hown in Figure 8), and analyzed behavioral and experiential apet of interation during tak ompletion. The 3 male and 3 female partiipant ame from a range of bakground, and all had normal (or orreted to normal) viion and no mobility impairment. The experiment wa onduted on an early prototype of Mirooft Surfae [20], uing the Nvidia PhyX gaming phyi engine [21]. Thi allowed ening of multiple fin- gertip a well a outline of hand and forearm. The experiment utilized a 3x3x2 within-ubjet (repeated meaure) deign. Eah partiipant worked through the three puzzle in eah of three interation tehnique: Joint, Proxie, and Partile. Pilot teting inluded a fourth ondi- further tion, Diret Fore, but thi wa dropped during teting (a explained below). For tehnique that do not intrinially upport olliion, a imple olliion model baed on kinemati objet wa applied, allowing intera- tion from the top and the ide of an objet. In addition, we hypotheized thatt the preene of viual feedbak howmight im- ing uer preiely where their input i applied prove the dioverability of eah tehnique. Viual feed- 73

8 bak of Joint wa repreented a red line drawn from the ontat point to the anhor point on the objet (thee diappeared if the joint wa broken); Proxie a red ube at eah enter point where ontat wa ened; and Partile a maller red ube per pixel in the ontour image. We ran eah tehnique with and without viual feedbak a our third independent variable. The tak etup (ee Figure 8) wa a follow. In Tak 1, eah of four phere and retangle were plaed exatly on mathing target; eah objet diappeared upon proper plaement. In Tak 2, an aortment of objet of different hape, ize, and mae were orted onto the left or right portion of the reen depending on their olor. In Tak 3, a ylindrial objet wa teered from a et tarting poition (far top right of photo) to a target (hown in red) by paing everal waypoint (hown in blue) without dropping the objet from a platform (whih aued the tak to retart). The tak were preented in the ame order to eah partiipant, wherea the order of interation tehnique wa ounterbalaned aro partiipant uing a Latin-quare deign. Experimentation ourred in two main phae (with viual feedbak of the input and without), preentation of whih wa, again, ounterbalaned aro partiipant. During the experiment, partiipant were not given any diret intrution, but had everal attempt to try out eah new puzzle. Partiipant performed eah tak twie (exluding any training), under experimental ondition, to provide an average ompletion time for eah ondition. Partiipant were interviewed informally after ompleting their eion. Tak 1 Tak 2 Tak 3 Figure 8. Tak 1: Exat poitioning of boxe and phere. Tak 2: Sorting by olor. Tak 3: Steering. Early Iue with Diret Fore Initially, the Diret Fore tehnique wa implemented by applying a mooth veloity at a given ontat point on the objet, omputed a a meaure of the diplaement between the ontat urrent and lat poition (i.e., kineti frition). Thi eemed a fair approximation for modeling urfae input a diret fore. However, our pilot quetioned the effiay of thi tehnique. Speifially, uer found it diffiult to omplete tak that involved aurately poitioning objet; i.e., moving and then topping an objet at the target loation. Moving the objet ould be performed reaonably, but to top it the uer needed to ounterat the motion in the oppoite diretion. Thi often led to exeive veloity applied in the revere diretion, auing objet to overhoot the target. Conequently, performane with thi tehnique wa o poor that we felt it needed no further evaluation. Baed on thee iue and feedbak from the pilot, we exluded thi tehnique from analyi. Initial Reult and Obervation Although thi wa only an initial exploration, we oberved many promiing interation and form of geture within the tudy. Uer eemed aware of the potential of thi new type of environment and exploited the phyi-baed ytem failitation of experimentation, and we oberved many new interation trategie. Kinemati Control and the Cure of the Single Finger Figure 9 how the ompletion time for all tak. Joint provide kinemati ontrol that loely mimi drag-anddrop behavior, and thu failitate eay poitioning of objet. Thi i refleted in the reult. After ome experimentation, there wa a moment when uer diovered that the objet wa under familiar kinemati ontrol. Uer ommented that my hand are like magnet or I an pre hard and tik my finger. Of oure, preure and magnetim were not fator at play here (in fat, pot-tudy interview revealed that partiipant were unure of the general priniple behind the Joint tehnique). Neverthele, uer performed the tak rapidly after diovering the objet wa omehow fixed to their finger. Figure 9. Tak ompletion time. FB denote ondition in whih viual feedbak of the uer input wa provided. However, the quantitative reult tell only part of the tory. During the tudy we alo oberved many limitation with the kinemati approah. The diovery of thi type of eentially drag-and-drop behavior in the Joint ondition led uer to predominately interat with a ingle finger and with a ingle objet at a time. Even rotation of an objet were predominantly undertaken uing a ingle finger [17]. Experimentation with multi-fingered or bimanual tehnique wa therefore rare in the Joint ondition. During informal interview, uer ommented that the ondition wa limited and le atifying than the other tehnique even though they performed the tak rapidly. Although it i too preliminary to draw ignifiant onluion, it doe ugget the need to meaure more than tak ompletion time when evaluating uh phyi-baed tehnique. Uer alo had a poor undertanding of how olliion were upported in a kinemati approah uh a Joint. We oberved many intane where aidental olliion aued by hit-teting on the ide a oppoed to the top of the objet would aue an objet to move away from the uer and aue a great deal of onfuion. Thi make u reviit whether a kinemati-plu-olliion model make any ene to the uer at all: Why indeed hould an objet only be tiky when you touh it top a oppoed to it ide? Thi atually led ome uer to infer that objet were magnetized in a way that upported both attration (when touhing the top) and repulion (when olliding with the ide). 74

9 Uing Feedbak to Go Beyond Kinemati Control A hown in the reult, feedbak did not play a ignifiant role in the Joint ondition, a one might expet given the familiarity of the approah. Feedbak played a more ignifiant role for Partile in Tak 1. After ome training time, uer diovered they ould interat with more than jut their fingertip. Bimanual upping and throwing and athing tehnique were devied to rapidly move objet to target poition (Figure 10). Thee trategie, and the general level of fine ontrol, enabled uer in the Partile ondition to obtain ompletion time omparable to more kinemati approahe. During interview, uer refleted poitively about the interation Partile afforded. However, thee type of ontour-baed bimanual interation ould not be utilized with Proxy objet although partiipant did try. In fat, in many ae, a hand geture on the urfae would be poorly approximated a a ingle proxy (the enter of ma of the ontat hape), auing objet to lip through a hand or auing other peuliar hitteting behavior. Multiple finger were ued to reorient boxe effetively, but overall, bimanual ontrol wa rare. Figure 10. Uing ontour of the hand to move multiple boxe (top left); provide a barrier to hange ball motion and poition moothly over target (top middle, right); throw and ath an objet from a greater ditane (bottom). While the drag and drop nature of Tak 1 learly favored kinemati ontrol uh a that offered by the Joint approah, Tak 2 offered a lear advantage to onurrent manipulation of multiple objet for rapid orting. A might therefore be expeted, ue of both Proxy and Partile tehnique, whih eemingly promoted multi-touh interation, led to fater ompletion time in thi tak (Figure 11). Figure 11. Two-handed and multi-fingered trategie adopted in the proxy and partile ondition. Coarely moving objet uing both hand (left), two-fingered rotation by applying torque to align a box (middle); fine-grained movement of two objet uing a ingle finger of eah hand (right). Coming to Grip with Non-planar Objet Another peifi trade-off in our deign wa that the rigid body ube in the Proxy ondition only provided an effetive mean for interating with flat objet. They provided little grip of pherial objet (or more omplex 3D mehe). Thi wa learly evident in the final tak where the Proxy ube truggled to keep the ylindrial objet under ontrol, a hown in Figure 8. In thi tak, we found uer often reverted to point-baed interation to ontrol the mall non-planar objet; the ue of ontour wa infrequent. However, our initial reult ugget that Partile till outperform Proxy objet for thee purely point-baed interation. Thi ugget that for enario where touh-only input i available, the Partile model ubume the ingle Proxy objet model. DISCUSSION The reult of the uer tudy and general experimentation ugget that while the more familiar kinemati approahe (omewhat inevitably) offer more preditable ontrol in ome ituation, the partile proxy approah an offer omparable performane while providing new mode of interation (uh a upping the ball in Figure 10). That our tudy partiipant were able to devie new manipulation trategie from limited feedbak and training i enouraging. With more time, we expet uer to further draw on their experiene with real-world manipulation. There are a number of way in whih our interative urfae imulation doe not math the phyi of the real world. In uggeting that we abandon familiar, kinemati point-baed ontrol in favor of trongly phyi-baed tehnique, an important onideration i whether uer are able to negotiate thee differene. Firt, while in the real world one might apply more or le fore to ontrol frition, our ytem ha no ene of how hard the uer i preing. When uing partile proxie, the amount of frition applied i intead proportional to the number of proxie applied to the objet, whih itelf i related to the urfae area of the touh. For example, a ingle finger moving aro an objet will apply le frition than multiple finger. Not urpriingly, thi ditintion wa generally lot on tudy partiipant, who often tried to pre harder to bring an objet under their ontrol. Similarly, our uer would ometime apply multiple finger to an objet when they wanted preie movement. Beaue of the inevitable impreiion of the imulation, the objet would move too unpreditably for very fine ontrol. In many of thee ae, it would have been better to ue a light (mall) touh rather than a full grip. Seond, graping a virtual objet by plaing ontat on either ide an be diffiult if not impoible in many of our tehnique. Suh manipulation require peritent reting ontat to be plaed on virtual objet. The partile-baed approah, in whih eah proxy i reated every frame, plae the proxy orreponding to a graping finger on top of the objet, thu defeating attempt to grap it. The ingle proxy objet approah ue peritent proxie, and o allow graping of an objet reting on the floor. It may be poible to extend the partile approah to allow proxie to perit at a given depth when it eem appropriate, or to explore a hybrid approah in whih both the partile and ingle proxy tehnique are ued imultaneouly. 75

10 Graping may be diffiult to repliate for more fundamental reaon. Virtual objet exert no ounterating fore againt the finger, o it i diffiult to know how hard the finger are preing on an objet. Graping an objet in order to lift it out of the plane may be hallenging to depit diretly on an interative urfae with a 2D diplay. Similarly, the enation of moving the hand aro an interative diplay urfae i the ame regardle of the imulated material, and whether the ontat exert tati or kineti frition. Some of thee problem may be addreed by improving the bai ening tehnique. For example, our ytem may be able to ene preure by interpreting the gray level of the input image a light touhe or heavy touhe. For proxy-baed tehnique, thi ould be implemented by hanging the ma of the proxy. New range-ening amera [26] providing per-pixel depth information may alo be appropriate. Per-pixel depth might be ued to ontrut a rih 3D model of the hand, opening new opportunitie in modeling graping behavior. It might alo ait in graping an objet in order to lift it up and plae it on another objet. Clearly one need not ompletely repliate the phyi of objet manipulation in order to ontrut ueful appliation exhibiting phyi-inpired behavior. The appropriate degree to whih the tehnique in thi paper are applied depend on the appliation. A game might naturally exploit detailed phyi throughout, while a graphial layout appliation might be very eletive. Joint ontraint provided by phyi engine may be ued to ontrain motion, for example, to eae alignment tak. While joint an be ued to imulate the real-world ounterpart of traditional GUI lider, dial, button, and the like (a uggeted by [10]), ome apet of traditional interation do not naturally lend themelve to a phyi implementation. Changing the ize of an objet dynamially, for example, doe not lend itelf to rigid-body imulation. CONCLUSION We have introdued a number of tehnique to inorporate interative urfae input primitive into a real-time phyi imulation. Our tehnique take advantage of the fidelity of ening provided by viion-baed interative urfae, with the goal of enabling in a virtual domain the range of objet manipulation trategie available to u in the real world. ACKNOWLEDGMENTS We thank Ni Villar for produing the aompanying video. REFERENCES 1. Agarawala, A. and Balakrihnan, R Keepin' it real: puhing the dektop metaphor with phyi, pile and the pen. CHI 2006, Barron, J., Fleet, D., Beauhemin, S., and Burkitt, T Performane of optial flow tehnique. Computer Viion and Pattern Reognition, D. Baraff, Analytial method for dynami imulation of non-penetrating rigid bodie, SIGGRAPH Computer Graphi, D. Baraff Fat ontat fore omputation for nonpenetrating rigid bodie. SIGGRAPH Computer Graphi, Chang, B.-W., and Unger, D Animation: from artoon to the uer interfae. UIST 93, Chang, F., Chen, C.-J., and Lu, C.-J A linear-time omponent labeling algorithm uing ontour traing tehnique, Computer Viion and Image Undertanding, vol. 93, no. 2, Dietz, P. and Leigh, D DiamondTouh: a multi-uer touh tehnology. UIST 2001, Dragievi, P Combining roing-baed and paperbaed interation paradigm for dragging and dropping between overlapping window. UIST 2004, Erleben, K., Sporring, J., Henriken, K., and Dohlman, K Phyi-Baed Animation. Charle River Media, In. 10. Fröhlih, B., Tramberend, H., Beer, A., Agrawala, M., and Baraff, D Phyially-baed manipulation on the reponive workbenh. IEEE VR Conferene 2000, Gonzalez, R., and Wood, R Digital Image Proeing: Third Edition. Prentie Hall. 12. Geißler, J Shuffle, throw or take it! working effiiently with an interative wall. CHI Ext. Abtrat, Groman, T., and Wigdor, D Going deeper: a taxonomy of 3D on the tabletop. Seond IEEE Internat'ional Workhop on Horizontal Interative Human-Computer Sytem, Han, J.Y Low-ot multi-touh ening through frutrated total internal refletion. UIST 2005, Hanok, M., Carpendale, S., and Cokburn, A Shallow-depth 3d interation: deign and evaluation of one-, two- and three-touh tehnique. CHI 2007, Hanok, M., Carpendale, S., Supporting Multiple Off-Axi Viewpoint at a Tabletop Diplay Seond IEEE International Workhop on Horizontal Interative Human- Computer Sytem, Kruger, R., Carpendale, S., Sott, S., Tang, A Fluid integration of rotation and tranlation, CHI 2005, Liu, J., Pinelle, D., Sallam, S., Subramanian, S., and Gutwin, C TNT: improved rotation and tranlation on digital table. Graphi Interfae 2006, Mander, R., Salomon, G., and Wong, Y.Y A pile metaphor for upporting aual organization. CHI 92, Mirooft Corporation. Mirooft Surfae NVIDIA Corporation. NVIDIA PhyX Reetz, A., Gutwin, C., Stah, T., Naenta, M., and Subramanian, S Superflik: a natural and effiient tehnique for long-ditane objet plaement on digital table. Graphi Interfae Roberton, G., Czerwinki, M., Laron, K., Robbin, D., Thiel, D., and van Dantzih, M Data mountain: uing patial memory for doument management. UIST 98, Ståhl, O., Wallberg, A., Söderberg, J., Humble, J., Fahlén, L. E., Bullok, A., and Lundberg, J Information exploration uing the pond. In Pro. CVE, Wilon, A PlayAnywhere: a ompat interative tabletop projetion-viion ytem. UIST 2005, Wilon, A Depth-ening video amera for 3D Tangible Interation. Seond IEEE International Workhop on Horizontal Interative Human-Computer Sytem, Wu, M, and Balakrihnan, R Multi-finger and whole hand geture interation tehnique for multi-uer tabletop diplay. UIST