Augmented Reality Integating Compute Gaphics with Compute Vision Mihan Tuceyan August 6, 998 ICPR 98
Definition XCombines eal and vitual wolds and objects XIt is inteactive and eal-time XThe inteaction and alignment of eal and vitual objects happens in 3D (I.e., not 2D ovelays). August 6, 998 ICPR 98 2
An Example Application Electic wies shown as augmentation in a physical oom. August 6, 998 ICPR 98 3
Motivation Claim: Augmented Reality can be deployed in much moe useful applications in the eal wold than vitual eality. August 6, 998 ICPR 98 4
Possible Applications Maintenance and Repai Ref: ECRC August 6, 998 ICPR 98 5
Possible Applications X Achitectue and Constuction Befoe Afte Images: coutesy of CICC poject at ZGDV, Faunhofe IGD August 6, 998 ICPR 98 6
Possible Applications X Inteio Design Ref: ECRC August 6, 998 ICPR 98 7
Possible Applications X Aid in constuction sites: X-ay view of a wall Image: coutesy of ZGDV/Faunhofe IGD and Holzmann AG August 6, 998 ICPR 98 8
Possible Applications X Aid fo assembly in ca manufactuing: instuctions fo two-handed scew fixing Image: coutesy of ZGDV/Faunhofe IGD, data coutesy of BMW August 6, 998 ICPR 98 9
Possible Applications X Aid fo assembly in ca manufactuing: leve inside the doo to be pushed to ight position Image: coutesy of ZGDV/Faunhofe IGD, data coutesy of BMW August 6, 998 ICPR 98
Moe Possible Applications X Compute Aided Sugey X Fashion Design o Appael Retail X Cicuit Boad Diagnostics and Repai X Facilities Maintenance X Industial Plant Maintenance X Road Repai and Maintenance August 6, 998 ICPR 98
Majo Components X Display subsystem X Rendeing subsystem X Video input and image undestanding subsystem X Inteaction subsystem X Registation and tacking subsystem August 6, 998 ICPR 98 2
Display Technologies See-though video display technology video Scan convesion and video keying Gaphics Wokstation gaphics Mixed Video and Gaphics August 6, 998 ICPR 98 3
Display Technologies X See-though Head Mounted Displays (see-though HMD) X Wold seen diectly by the eye with gaphics supeimposed Gaphics Wokstation August 6, 998 ICPR 98 4
Rendeing subsystem X VR technology 3D eal-time endeing 3D inteaction head and camea tacking August 6, 998 ICPR 98 5
Vitual Camea X 3D gaphics endeed though a vitual camea X Realistic augmentation (with egistation and coect optics) ==> vitual camea matches eal camea o optics of human eye. August 6, 998 ICPR 98 6
Coodinate Systems in seethough video Camea Coodinates A Pointe Coodinates D Wold Coodinates E G F Camea Mak G Pointe G 3 2 Mak F 3 Object Mak C Tacke Tansmitte Coodinates Object Coodinates August 6, 998 ICPR 98 7 F 2
Calibation X All the coodinate tansfoms in the pevious slide need to be estimated. X This is done though calibation. August 6, 998 ICPR 98 8
Image Calibation X Scan convete can access an abitay egion of the sceen to be mixed with gaphics. X Modeled as 2D tanslation and scaling Sceen August 6, 998 ICPR 98 9
August 6, 998 ICPR 98 2 Image Calibation Image of two known points displayed and the esult is captued x c A B y Compute Geneated image A B Gabbed image y y x x t y k t x k c + = + = Calibation estimates k x, k y, t x, t y by equations A y A y A x A x A B A B y A B A B x y k t x k c t y y k x x c c k ~ ~ ) /( ) ( ~ ) /( ) ( ~ = = = =
Camea Calibation X Estimates the initial pose of the camea: R: otation matix T: tanslation X Estimates the camea intinsic paametes: f: focal length (, c ): image cente ( sx, sy ): scale factos Camea being calibated Calibation jig August 6, 998 ICPR 98 2
Camea Calibation (continued) X Many camea calibation methods exist in compute vision liteatue: R. Tsai Weng, Cohen, Heniou X The estimated camea paametes ae used in the vitual camea with which 3D gaphics ae endeed. August 6, 998 ICPR 98 22
Camea Calibation (continued) Points ae picked fom calibation jig whose 3D coodinates ae known The model of calibation jig is endeed using a vitual camea with the estimated paametes. The endeed gaphics is supeimposed on the image of calibation jig. August 6, 998 ICPR 98 23
Camea Calibation (continued) ( ( s f u, c, s R = T = v : focal ) :image cente ) :scale factos in x and y (o u and v) [ ] ij t t t 2 3 length : otation matix fo the camea : tanslation vecto fo camea diections August 6, 998 ICPR 98 24
August 6, 998 ICPR 98 25 Camea Calibation (continued) 3 33 32 3 2 23 22 2 3 33 32 3 3 2 t z y x t z y x f c c f s c c t z y x t z y x f f s v v u u + + + + + + = = + + + + + + = = Camea Equations given by: and columns ) : image coodinates in ows, ( c
Camea Calibation (continued) X Collect image data points ( i, c i ) coesponding to known 3D calibation points (x i,y i,z i ) X These coodinates ae coected using the esult of image calibation. X Substitute into camea equations in pevious slide ==> gives 2 equations pe calibation point. August 6, 998 ICPR 98 26
August 6, 998 ICPR 98 27 Camea Calibation (continued) X Vaiable substitution to lineaize camea equations: 3 6 3 2 5 3 4 t w t c t f w t t f w c f f v u v u = + = + = = + = + = 3 3 3 2 2 3 R W R R W R R W = = 3 2 3 2 i i i i T T T w w w W R R R R whee
August 6, 998 ICPR 98 28 Camea Calibation (continued) = n n n n n n n n n n n n n n n n n n n n c z c y c x c z y x z y x z y x c z c y c x c z y x z y x z y x A whee Solve the homogeneous equation AW =
Pointe calibation X 3D pointe is implemented with magnetic tacke. X The tacke coodinate system and the pointe tip needs to be elated to the wold coodinate system. X This is done though a pointe calibation pocess. Pointe Coodinates Tacke Tansmitte Coodinates Pointe Mak coodinates Wold Coodinates August 6, 998 ICPR 98 29
Tacke and pointe calibation X Calibation done by picking points fom the calibation jig. Known points picked Some points ae picked with diffeent oientations August 6, 998 ICPR 98 3
Pointe Calibation The tip of the pointe is given by the equation: p = p + I I I W R R R M M M M W T R Reading the pointe at n diffeent oientations, we get the equation 2 p p M p p p = p M M T n M n Solve the ovedetemined system fo pt and p W 2 p T p W Receive coodinate system oientation: R M p M Tacke Tansmitte Coodinates August 6, 998 ICPR 98 3
Tacke tansmitte calibation X Estimate position of tansmitte in wold coodinates X Read thee known points on calibation jig. X The oigin of the wold, p, with espect to the tansmitte is solved fom p afte the otation R MJ is estimated August 6, 998 ICPR 98 32 p L J P w = M + J J R M J p
Tacke Tansmitte Calibation X The otation matix R MJ is estimated by fist computing the axes: x = p W p L W J z = pw p z x y z x P W J = T L J P x z R M J = [ y ] x z August 6, 998 ICPR 98 33
Registation X Aligning 3D models of objects with thei eal countepats. X Alignment eos must be extemely small. August 6, 998 ICPR 98 34
Registation (examples) Example applications whee egistation is used A 3D model of the table is egisteed with the physical table to... obtain the effect of occlusion by using the geomety of the 3D model August 6, 998 ICPR 98 35
Registation (Examples) A 3D model of the engine is egisteed with the physical engine to... label the pats of the engine using the pats knowledge encoded in the 3D model. August 6, 998 ICPR 98 36
Registation Methods X Image based landmaks fo computing object pose. X 3D pointe based fo computing object pose. X Automatic egistation of objects in 2D images using compute vision techniques. August 6, 998 ICPR 98 37
Image Based Landmaks X Camea paametes known fom camea calibation. X 3D coodinates of landmaks known fom object model. X 2D image coodinates of landmak points picked manually. August 6, 998 ICPR 98 38
August 6, 998 ICPR 98 39 Image Based Landmaks X Plug the known values into the camea equations: X Solve fo tanslation and otation unde the constaint of othonomal otation matix. ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( 2 23 22 2 3 33 32 3 3 2 3 33 32 3 = + + + = + + + t f z f y f x f t c c z c c y c c x c c t f z f y f x f t z y x v i v i v i v i i i i i i i u i u i u i u i i i i i i i
Image Based Landmaks X Solution obtained by minimizing Ax 2 + α R T R I 2 whee x is the vecto of unknowns. August 6, 998 ICPR 98 4
3D Pointe Based Landmaks X Calibated 3D pointe device, X 3D coodinates of landmak points known fom the object model, X 3D landmak points picked on the physical object with the pointe. August 6, 998 ICPR 98 4
3D Pointe Based Landmaks X The picked 3D landmak points and 3D model landmaks ae elated to each othe with a igid tansfomation: p whee p p W = Rp L + T fo i =,, i i W i L i : : wold coodinates landmak coodinates n August 6, 998 ICPR 98 42
3D Pointe Based Landmaks X Solution by minimizing equation: p W i Rp L i T 2 + α R T R T 2 August 6, 998 ICPR 98 43
Automatic Registation X Compute Vision techniques fo egisteing objects with thei images and computing object pose X Featues detected automatically and matched to model X Less geneal X Less obust August 6, 998 ICPR 98 44
Registation Pitfalls X R matix in the solution to these equations may not be a valid otation matix. X One possible appoach is to fomulate the equations epesenting the otation as a quatenion instead of a 3x3 matix. August 6, 998 ICPR 98 45
Tacking subsystem X Many tacking system possibilities Magnetic tackes Mechanical tackes Optical tackes Ultasound tackes Vision based tackes August 6, 998 ICPR 98 46
Magnetic tackes X Objects ae tacked by attaching a eceive of magnetic tackes onto the objects. X The data ead fom the magnetic tacke is used to update the object-to-wold igid tansfomation. X This equies a calibation pocedue (as pat of object egistation) by which the object-mak-toobject tansfomation is initially estimated. August 6, 998 ICPR 98 47
Vision Based Tacking X Camea tacking X Object tacking One implementation using landmaks put in the envionment. Camea motion is computed automatically. August 6, 998 ICPR 98 48
Vision Based Camea Tacking X Envionment modified with landmaks whose 3D coodinates ae known X Landmaks detected automatically X camea pose extacted fom cone coodinates of squae landmaks simila to camea calibation pocedue. August 6, 998 ICPR 98 49
Vision Based Camea Tacking X Landmak squaes in the envionment ae detected (segmented) using a wateshed tansfomation algoithm Oiginal image Wateshed tansfomation Results of inside opeation fo egions of wateshed tansfomation August 6, 998 ICPR 98 5
Vision Based Camea Tacking X Landmak squaes contain ed squaes at known positions which encode the identity of the model squaes Red squaes ae baely visible in the geen and blue channels. Black squaes segmented in the geen channel. Red channel is sampled at locations whee ed dots should be w..t. the landmak bounday giving the id of the squae. August 6, 998 ICPR 98 5
Vision Based Camea Tacking Squaes detected and oom scene augmented with a wall endeed with pope pespective August 6, 998 ICPR 98 52
Vision Based Camea Tacking X Motion model assumed: p = v + ω c ( p c) whee c is the cente of mass of the object, p is a point on the object,v c is the tanslational velocity at c, and ω is the angula velocity aound an axis though c. August 6, 998 ICPR 98 53
Vision Based Camea Tacking X Extended Kalman filte employed fo optimal pose and motion estimation. August 6, 998 ICPR 98 54
Vision Based Camea Tacking Results August 6, 998 ICPR 98 55
Vision Based Camea Tacking X Example video clip showing camea tacking with augmentation of the scene August 6, 998 ICPR 98 56
Vision Based Camea Tacking X Occlusions of landmaks toleated as long as at least two landmaks ae completely visible. August 6, 998 ICPR 98 57
Vision Based Tacking X Objects can also be tacked using landmaks on them that can be detected automatically. X Othe vision-based tacking methods may also be used such as optical flow. August 6, 998 ICPR 98 58
Hybid Tacking X Hybid methods fo tacking have been developed fo augmented eality UNC eseach: combination of magnetic tackes and vision based landmak tackes Foxlin: combining inetial tackes with ultasound. August 6, 998 ICPR 98 59
Inteaction of Real and Vitual Objects X Occlusions X Inteactive queies (e.g., 3D pointe) X Collisions X Shadows and lighting effects August 6, 998 ICPR 98 6
Inteactions of Real and Vitual Objects: Occlusion X Method : model-based egisteing a model with the physical object endeing the object in black so gaphics can be choma-keyed use z-buffe to occlude August 6, 998 ICPR 98 6
Inteaction of Real and Vitual Objects: Occlusion X Method 2: geomety extaction compute dense depth map use the computed depth map in z-buffe to occlude vitual objects. August 6, 998 ICPR 98 62
Inteaction of Real and Vitual Objects: Occlusion X Dense depth map can be computed using steeopsis (o othe ange sensos) Left Image Right Image August 6, 998 ICPR 98 63
Inteaction of Real and Vitual Objects: Occlusion Computed Dense Depth Map (dake w close) Depth Map used as z-buffe to obtain occlusion effect August 6, 998 ICPR 98 64
Inteaction of Real and Vitual Objects: Collision X A synthetic object placed in a eal scene must give the peception of physical inteaction with its envionment w collision detection is an impotant poblem. August 6, 998 ICPR 98 65
Inteaction of Real and Vitual Objects: Collision X Requiements: geomety of the synthetic objects is known geomety of the scene is known X Classical methods of collision detection can be utilized based on geomety intesection. Ref: John Canny wtime consuming August 6, 998 ICPR 98 66
Inteaction of Real and Vitual Objects: Collision X Method 2: use z-buffe techniques Some applications ae sufficiently constained that they can cheat and still have a ealistic effect. Scene geomety can be used to build a z-buffe, which then can be used to check fo simple collision tests with the synthetic objects. Ref: Been et al. August 6, 998 ICPR 98 67
Illumination X Realistic mixing of synthetic objects with images of the scene also involve: Rendeing the synthetic objects with the same illumination as the actual scene w extact illumination souces in the scene Having objects cast shadows on each othe in a coect manne w extact illumination as well as geomety of the scene. August 6, 998 ICPR 98 68
August 6, 998 ICPR 98 69 Illumination X The effect of illumination is summaized by the equation: + + = ω ω ω ω π d H N c L c k d L N c L c k c L c k c L n s d ia a pixel ) )( ( ) ( ) )( ( ) ( ) ( ) ( ) ( the colo channel }is,, { whee B G R c
Illumination X Most illumination estimation methods utilize some fom of shape-fom-shading fomulation fom the diffuse component of the pevious equation. August 6, 998 ICPR 98 7
Illumination Light Souce L Nomal N Viewe V I = ρλ( N L) X ρ : albedo of the suface X λ : flux pe unit aea pependicula to incident ays X Ref: Pentland PAMI 984 August 6, 998 ICPR 98 7
Illumination X In a local homogeneous aea of the suface, albedo and illumination change little X This assumption gives us: di = d( ρλ( N L)) = ρλ( d N L) + ρλ( N d L) = ρλ( d N L) August 6, 998 ICPR 98 72
Illumination X Assumption: changes in suface oientation ae isotopically distibuted. X Look at the effect of illuminant diection on di, the mean value of di. di = ρλ( d N L) = ρλ( x L d x N + y L d y N + z L d z N ) ( d x ( x L N, y, d y L, z L N, d z N ) : mean change in ) :light souce diection d Nmeasued along diection ( dx, dy) August 6, 998 ICPR 98 73
Illumination X Assuming change in suface nomal is isotopic, then and d x d y d z d N N N I is is = popotional popotional to to dx dy = k( x dx y dy) L + L August 6, 998 ICPR 98 74
Illumination We can estimate illumination diection fom the last equation n by measuing the mean of di along a numbe of diections o and setting up a linea egession fomulation to estimate light souce diection August 6, 998 ICPR 98 75
Illumination X Integated methods fo estimating illumination diection with shape-fomshading also exist Ref: Hougen + Ahuja, ICCV 93, Belin X Moe ecent woks of illumination extaction: Ref: Walte, Alppay, et al, Siggaph 97 August 6, 998 ICPR 98 76
Illumination and Inteaction of Real and Vitual Objects: Shadows X Shadows cast by eal objects on vitual objects and vice vesa incease the ealism in some applications scene and object geomety should be known illumination model should be known then by eendeing the scene one can compute the shadows in the mixed eality scene Ref: Founie August 6, 998 ICPR 98 77
Calibation-fee Augmented Reality X Display and ovelay of gaphics is done without extaction of camea paametes. X Basic opeation is epojection: Given: Fou o moe 3D points,» the pojection of all the points in the set can be computed as a linea combination of the pojection of just fou of the points. X Ref: Kutulakos & Vallino in TVCG Jan-Mach 998. August 6, 998 ICPR 98 78
August 6, 998 ICPR 98 79 Calibation-fee Augmented Reality X Repojection popety p b b 2 b 3 ζ : χ ψ Affine viewing diection Affine bases points + p ae the fiducial points tacked that define the affine epesentation = 3 2 3 2 z y x z v v v v v v v u u u u u u u w v u p T p p b p b p b p p b p b p b ς Affine coodinates of p on an object
August 6, 998 ICPR 98 8 Calibation-fee Augmented Reality X Affine coodinates of a 3D point p can be computed fom thei pojections along two diffeent viewing diections. = 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 3 2 3 2 3 2 z y x v v v v v v v u u u u u u u v v v v v v v u u u u u u u v u v u p p b p b p b p p b p b p b p p b p b p b p p b p b p b p p p p Solve fo x, y, and z Affine coodinates Known fom coodinates in two images
Calibation-fee Augmented Reality X The affine epesentation of a 3D object can be used by tacking 4 fiducial points acoss fames and using thei image coodinates to compute epojection of the othe points on the objects. August 6, 998 ICPR 98 8
Futue Reseach Topics X Calibation of see-though HMD s Optics ae diffeent (human eyes ae in the loop) Thee is no explicit image fom which points ae to be picked diectly. Calibation must be done using some othe, possibly inteactive technique. August 6, 998 ICPR 98 82
Futue Reseach Topics X Inceased accuacy and automation in object egistation, and tacking (camea and/o object) X New display technologies fo example, pojection of gaphics onto eal scenes August 6, 998 ICPR 98 83
Resouces X My Web page addess: http://www.cs.iupui.edu/~tuceyan/ar/ar.html X Augmented Reality Page by Vallino http://www.cs.ocheste.edu/u/vallino/eseach/ar Many links to othe places fom hee. X Faunhofe AR page: http://www.igd.fhg.de/www/igd-a4/a/ X MIT AI Lab image guided sugey page: http://www.ai.mit.edu/pojects/visionsugey/sugey_home_page.html August 6, 998 ICPR 98 84