Augmented Reality: projectors

Transcription

1 Augmented Reality: projectors by Michael Vögele 1. Introduction 1.1. Brief overview on research areas 2. History of displays/projectors 3. Smart Projectors 3.1. Description 3.2. Areas of application 3.3. Core technological realization Undistorted projection Color correction Lamberts Law Neutralized texture 3.4. Applied technological tools 3.5. Smart Projectors in habitual environments Calibration Multiple Viewers 3.6. Multiple projectors 4. Enabling View-Dependent Stereoscopic Projection 4.1. Prerequisites 4.2. Innovation compared to smart projectors 4.3. Core technological realization 4.4. Areas of application 5. Dynamic Shader Lamps 5.1. Description 5.2. Shader Lamps Painting on Movable Objects 5.3. Areas of application 5.4. Technical aspect Projector arrangement Tracking + Calibration Choice of objects The brush 5.5. Problems waiting to be resolved 6. Seamless integration of VR into habitual workplaces 6.1. Brief Description Innovation + technology 6.2. Areas of application 7. References 1. Introduction Through the course of the 20th century television has played an important role in the entertainment industry. TV sets were continuously improved making them bigger, improving their refresh rate, minimizing power consumption and cutting the price. Todays flat panel displays however are reaching their limits. Projectors may just be the next evolutional step in displays. Not only could projectors push the boundaries given by flat panel displays, in addition they could be used to incorporate augmented reality into our every day lives. Hence the following document is aimed at presenting these posibilities using specific research projects for demonstration purposes Brief overview on research areas The core of this document is comprised of research projects aimed at taking video displays to the next level. The remaining projects demonstrate the posibilities beyond mere video displays. The following projects will be presented: Smart Projectors Stereoscopic Projection Shader Lamps integration into workplaces 2. History of displays/projectors The idea of projectors first arose in the 15. century. At the time projectors consisted primarily of a lantern and a lens. In 1825 limelight projectors were discovered. These projectors used a limestone as light source (limestones glow if lighted by an oxy-hydrogen flame). In the end of the 19th century limelight was replaced by electric light. With the arrival of digital systems projectors were adjusted to work with digital data. They first became usefull for presentations primarily in big companies. From this point on projectors were improved much the same way computers were. They became smaller, easier to handle and better in quality. Today projectors are expanding into new areas such as television, entertainment, information... Furthermore they are evolving to present us with new posibilities such as three dimensional projections.

2 3. Smart Projectors 3.1. Description Smart projectors are video projectors capable of projecting onto any aurface. Hence they eliminate the necesity of a large screen solving the main problem with projectors for habitual workplaces. The task of projecting onto arbitrary surfaces requires the use of sensors to scan the desired canvas. In theory all sorts of sensors can be used, smart projectors however concentrate on the use of cameras. At its core a smart projector is any projector that analyzes its environment before projecting Areas of application Smart projectors could be used in a wide variety of areas. Smart projectors could for instance be used as a means of applying information directly onto objects without altering them physically. This could be interesting for museums if they want to project information onto ancient artefacts. Most interestingly however smart projectors may drastically enhance the use of projectors in every day homes. They are capable of using any surface for a large screen experience. In the not too distant future smart projectors may even become a part of unconspicuous actions such as searching for a book or doing routine work at ones desk. Here they would circle the requested book in a shelf or project virtual objects onto your desk (see 6). Unlike large television sets smart projectors also have a benefit in that they are fairly small and can therefore easily be transported Core technological realization Undistorted projection The projector realizes undistorted projection by processing incoming video signals according to parameters gained by a one time calibration. The calibration process happens only once for a specific screen and camera position. Hence it is currently not possible to project onto a moving canvas. The projector is calibrated by emiting specially structured light onto the screen. The camera then transfers a snapshot of the lighted area to the smart projector which analyzes the given data to extract the desired parameters. The system usually only concentrates on a specific set of pixels. The projector emits these pixels as if projecting onto a flat screen. By comparing the pixel positions that were projected to those actually seen by the camera a displacement map that will map video pixels onto the desired screen can be constructed. In order to achieve an acurate displacement map multiple different patterns of light are emited and analyzed. The resulting data is then combined. In order to synchronize projection with the camera it is vital for the projector to know how long it takes until the projector receives the image taken by the camera. This time is initially measured by emiting a pattern and saving the time it takes until the image arrives through the camera. The entire process takes aproximately 28 seconds. If the surface is very rough more pixels must be analyzed which results in longer calibration times or the need for more advanced algorithms. The process summarized: 1. Positioning of camera and projector 2. Measuring camera latency 3. Mapping projector to camera for multiple types of structured light 4. Construction of a displacement map Color correction Up to this point we have only corrected the geometry of our screen. The screens color however still interferes with the color provided by our projection. In order to solve this problem one has to distinguish between surface types. While the color of some surfaces is altered by specular, reflection etc. effects the color of some surfaces is solely defined by its diffuse color. To this point research on smart projectors focuses only on the latter due to their simplicity. These surfaces are known as Lambertian surfaces. If however a surface absorbs a crucial part of the light spectrum color correction as well as geometry correction will fail Lamberts Law In order to understand Lamberts Law it is crucial to understand Lambertian surfaces. The color of a Lambertian surface can be acurately defined by a few decisive parameters: 1. surface s material color (M) 2. light color 3. intensity that leaves the source (I) 4. distance to the surface (r) 5. incidence angle of light rays (a) Lambertian surfaces are diffuse reflectors which means that light hitting the surface is reflected in all directions with the same intensity. The reflected intensity is dependant on the angle between the light vector and the surfaces normal. These attributes are taken into account in Lamberts Law: R = I*cos(a)/r^2*M R aproximates the diffuse reflection.

3 Neutralized texture In order to acurately display an image the projector must neutralize undesired color and light intensity. This is done using Lamberts Law. We start by adjusting Lamberts Law in such a way that it takes the environmental lighting into account. R = I*cos(a)/r^2*M+EM E holds the intensity and color of the environmental light. Solving the equation for I gives us the desired projection intensity. I = (R-EM)/(M*cos(a)/r^2) The last question remaining is how the parameters are acquired. The returned intensity R is given by the original image. EM can be aproximated by turning off the projector and using the camera to measure the light intensity. The result is a value proportional to EM which can be used to aproximate the value. M*cos(a)/r^2 can be aproximated in much the same way except that this time the projector projects using full light intensity (I = 1) Applied technological tools Up to this point smart projectors use devices accessible to the public. The basic configuration consists of a projector, a camera, some from of microcontroller and a graphics card. All data is saved in textures in order to make use of pixel shader technology found in modern graphics cards. The GPU does all the necessary calculations and then repeatedly uses a displacement texture to transform the input image into the desired projection on a pixel by pixel basis. The detail levels given by the camera and the projector are still a major weakness for smart projectors Smart Projectors in habitual environments Calibration Before usage the user must calibrate the smart projector. This is done by placing the camera as close to the optimal viewing position ( sweet spot ) facing the screen. The projector does all necessary calculations discussed in chapter 3.3. This process takes about 28 seconds which is very acceptable for every day use. Once calibrated the camera can be detached and it is not needed again until the projector needs to be recalibrated Multiple Viewers The calibration process optimizes the projection for one specific viewing position, the sweet spot. This means that people sitting next to eachother may see a differing degree of distortion in the screen. If however one refrains from using extreme surfaces such as a corner of the room the difference in perception is rather small Multiple projectors Up to this point we still face a serious problem. On some surfaces the projector may cast shadows causing the emitted light to be absorbed in these areas. This particular problem can be solved using more than one projector. Having multiple projectors means having to achieve the desired light intensity by evenly distributing it over the given projectors. This leads to the following adaption of Lamberts Law: R = EM + I1*F1*M + I2*F2*M + + IN*FN*M. Assuming every projector emits light with the same intensity we get: Ii = (R - EM)/( F1*M + F2*M + + FN*M) Multiple projectors also enable us to split the screen into tiles projected by different projectors. In this way we can create large scale screens in high resolution. 4. Enabling View-Dependent Stereoscopic Projection 4.1. Prerequisites View-dependant stereoscopic projection uses the posibilities given by smart projectors and takes the innovation to the next step Innovation compared to smart projectors Using the capabilities of smart projectors enables us to acurately view a projection on an arbritrary surface from one fixed location, the sweet spot. The next step is to allow the sweet spot to be moved in real time. Using this capability the sweet spot can be moved according to the viewers tracked location allowing the user to acurately see the projected image from any angle. Having tracked the spectators position the projector can use this information to determine what to display on

4 screen creating the illusion of a three dimensional display Core technological realization Up to date two methods have proven usefull in achieving the desired effect. The first method is purely image based. Instead of having just one camera to calibrate the projector multiple cameras are used to cover a variety of possible viewing positions. Using the viewers tracked position the projector can calculate the projection parameters by acurately weighting the cameras parameters. This aproach retains the benefits smart projectors had in that the set up is completely independant from the chosen screen. 5. Dynamic Shader Lamps 5.1. Description 5.2. Shader Lamps The aim of research on dynamic shader lamps is to create movable objects with virtual texture. The virtual texture is projected onto the given physical object using multiple projectors. Shader lamps are projectors used to paint the surface of a physical object. The given object is white in reality, it s texture is altered by projecting the desired images onto its surfaces. Basic shader lamps only work for stationary objects Painting on Movable Objects An interesting aspect of shader lamps is that they allow the texture of an object to be manipulated easily. This allows for the posibility of painting on an object Areas of application A second aproach is more geometry based. It either tries to analyze the geometry of the screen and save it as a three dimensional modell or it uses predesigned modells. Based on the modell and the viewers position the projection is then calculated in real time. This method is however strongely dependant on the quality of the modell and is therefore difficult to use for arbitrary surfaces Areas of application The additions given by stereoscopic view dependant projections can be used in the same areas the smart projectors can be used. Three dimensional projectors could enhance your every day lives at home, or work. A very likely area of application could also be a museum. Shader lamps could become usefull in video conferences. The fact that all texture data of objects is given virtually enables the data to be distributed between participants. If one person alters an objects texture an exact replica of the object in other areas would show the same texture. Further applications may include the cosmetics industry, art galleries, clothe shopping or childrens toys Technical aspect Projector arrangement For shader lamps to function correctly it is sufficient to have two projectors. The quality of this setup will however depend highly on the geometry of the object used. Some objects may cast shadows using only two projectors.

5 Tracking + Calibration The object is tracked using a sensor attached to the object. Using this sensor one can easily acquire the transformation matrix of the object. Multiplying the matrix by each vertex of the object yields the object vertices in world space. These are then used to acurately project onto the object. The projectors need to be calibrated once. During this process a transformation matrix is created which transforms points in world space into projector space Choice of objects In theory any object can be used. However using objects with Lambertian surfaces offers the same advantages as with smart projectors. Keeping the geometry simple is also beneficial in order to prevent shadows and in order to keep the data size small. Komplex modells also require the use of automated aproaches in gaining the geometry data. Here the same techniques as with stereoscopic projection can be used (see chapter 4.3.) The brush The brush is used to paint on the object. How much the brush affects the object depends solely on the brush s distance from the object. This is defined in the brush function: B(X) = 1 - (r/r)^2, r as distance; R as brush radius The color at a specific position x on the surface of the object is calculated using alpha blending: Color(x) = Color(x)*(1-B(X))+BrushColor*B(X) 5.5. Problems waiting to be resolved There are still a couple of limitations for dynamic shader lamps. - The sensors have fairly hight latency which disturbs the synchronisation between moving the object and projecting the texture. - The sensors have a fairly small range which limits the size of the entire setup. - As seen with smart projectors shadows cause problems with shader lamps as well Innovation + technology This aproach combines virtual objects with real objects. Both are manipulated with ones hand, real objects directly and virtual objects by means of gestures. An important technological aspect that is not covered in previously mentioned research areas is the challenge of tracking a users hand motion Areas of application A wide variety of applications are imaginable. Virtual notes could be made on ones desk, these could be easily archived. The technology could be used as a search function in which projectors mark real books that one is looking for. Rooms could have virtual pictures which adapt to whoever is presently in the room. Visualizations could aid when trying to repair something. 7. References Bimber, O., Emmerling, A., and Klemmer, T., Embedded Entertainment with Smart Projectors Oliver Bimber, Enabling View-Dependent Stereoscopic Projection in Real Environments, Submitted to the International Symposium on Mixed and Augmented Reality (ISMAR 05), 2005 Bandyopadhyay, D.; Raskar, R.; Fuchs, H., Dynamic Shader Lamps : Painting on Movable Objects, IEEE and ACM International Symposium on Augmented Reality, pp , October 2001 Bimber, O., Encarnação, L.M. and Stork, A., Seamless integration of virtual reality in habitual workplaces, Journal for Industrial Science, Munich University of Technology, vol.55, no.2, pp , Seamless integration of VR into habitual workplaces 6.1. Brief Description This research topic deals with incorporating virtual projections into our every day lives. Emphasis is placed on the use of projectors in order to add virtual objects to the desk environment.