Augmented reality Project in computer sience: Report

Transcription

1 PROJECT REPORT Augmented reality Display of possibilities of augmented reality using Java and C++ Marcus Agelii & Gösta Agerberg Supervisor: Maya Neytcheva & Sergio Courtade Project in computer sience: Report January 2013 DEPARTMENT OF INFORMATION TECHNOLOGY

2 Abstract The purpose of this project is to explore augmented reality and investigate its possibilities. Another target was to have software which ran on a mobile device. In this report we summarize the conducted study on the current progress in the field and describe two applications, for Android, developed under the project work. The applications demonstrate what augmented reality can be used for. The applications are written in Java and C++.

3 Contents 1. Introduction What is augmented reality... 4 Markers... 4 Augmented reality - early history and current development... 5 Augmented reality for mobile devices... 6 Android applications Project description and aims Problem description... 7 Limitations... 7 Goals Workflow and implementation Results and conclusions Logotype app Virtual buttons app Conclusions Discussion Difficulties Possible developments of our apps Possibilities of Augmented Reality Acknowledgements References

4 1. Introduction What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The best way to explain this further is by an example: Imagine that you are looking at a screen; the screen shows a camera feed of today s newspaper. When the camera comes to a certain picture in the newspaper, instead of just showing that newspaper with the picture, it instead shows a video taking the picture s place in the newspaper. The effect is achieved by enabling the program that detects the picture to search through its database looking for what actions to take when it finds that exact image. This specific program has videos as actions, and uses it by overlaying the image by the video. Figure 1: Picture in a newspaper being replaced with a video using augmented reality Markers To detect something in reality an AR application needs certain input; this is usually from a camera but could also be from other sources such as GPS or accelerometers. If the AR application uses a camera feed as an input and recognizes an object using image recognition software, this object is called a marker. Often specific images are used as markers but it could also be real objects such as a car or a building. There is also an AR technology called markerless based AR, instead of trying to recognize specific markers from a camera feed, this technology tries to do a mapping of the surrounding environment and uses this map to augment reality. 4

5 Augmented reality - early history and current development AR as a concept and idea has been around for many years but it s only in the recent years that technology has been made small and fast enough to make AR available for everyday use. The first AR system was developed in the 60 s but the term AR was not coined until 1990 by a Boeing researcher called Tom Caudell. 1 One of the first uses of simple AR technology was by pilots in fighter jets, where the pilot had a HUD (Heads Up Display) showing altitude, velocity and other useful flight data in their main field of view. This can however be discussed as being real AR but when this technology evolved into also showing waypoints and cues over targets (Target designators) this becomes more of what people see AR as today. In recent years as mobile phones and handheld devices have been made cheaper and more powerful, the potential of AR applications have increased significantly. Today there are many apps available to download from both Google play and Apples App store. The application field of these apps ranges from navigation to entertainment, home furnituring etc. There is however still a lot of developing needed before these apps surpasses the usefulness and the reliability of other methods and apps not using AR. Figure 2: A HUD display in a fighter jet. 1 Kangdon, Lee; 5

6 Augmented reality for mobile devices AR right now is optimal for mobile devices and especially for smartphones. The smartphones have the ingredients needed for the AR to work, a camera, a CPU and a screen, all in the same small package. Another feature that makes smartphones very suitable for implementing AR apps is their accessibility both to consumers and to developers. It is easy to use a smartphone and it is easy to develop software for a smartphone. The smartphone is also very mobile and as such very easy to move around an augmented object. Since the smartphones also have other detectors such as accelerometers and GPS, these can also be used when handling AR. Android applications Android is the name of an operating system primarily used for mobile and tablet devices but also in other devices such as cameras and watches. 2 Android is based on Linux but have been further developed by Google to better suit smartphones and tablets (such as energy management). Android is open source and primarily developed by Google, but due to its open source nature many carriers and smartphone manufacturers have adapted Android to fit their products. For Android there is a wide variety of third-party applications that can be installed on devices running Android. These applications (apps) are either distributed by Google s own store called Google play or by other available stores such as Amazon store, or by third party websites since it s possible to download.apk files to an android device and install them manually. To develop applications for Android, Google have released an Android Software Development Kit (SDK) and the applications are developed using Java. This software kit includes all that is needed to start developing applications for Android including a debugger, software libraries and tutorials. There is also a native development kit available for applications or extensions in C or C++. The official Integrated Development Environment (IDE) supported is Eclipse 3 using the Android Development Tools plugin

7 2. Project description and aims Problem description The project description for our project was rather general and did not contain any specific details and limiting conditions. Our supervisor gave us the AR for mobile devices topic and ideas in this topic to investigate and let us set the goals and limitations by ourselves. The ideas to focus on were basically three; the image analysis involved in reading the images, the creation and reading of information from the database, or the projection of information. Therefore our first task was to actually come up with a more specific project description. An advice we got from our supervisor was to not invent the wheel again and by having this in mind we decided to focus on already built up SDK:s and investigate how these worked and what could be done with them. Since we didn t want this to become merely a literary study, we decided that if possible, we wanted to create an application that would display the possibilities of AR in an easy way. In the short description we got, it was preferred that we would work on mobile devices so we decided that our applications target would be a smartphone of some kind. Limitations We had to set up our own limitations, but since neither we nor our supervisor had any experience in the subject we had no idea how to set them up initially. We did some research to find a suitable library to work in, and then some more research to see what this library could do for us, and what we could do with it. This resulted in that we limited ourselves to the third alternative for specialization the displaying of information. We also narrowed our project down to using a library called NyARToolkit 4 and working in Windows. The last two limitations proved to be unnecessary, since we early in the project found out that the limitations of NyARToolkit were too strict to fit what we were interested in doing. More research paved way for a library called Vuforia; this library had more functions built in, was up to date and offered the possibility to more easily write an application for a mobile device. This meant that our new limitations sent us towards an Android application, and also the new experience in programming for a mobile device. Goals To merge our project description together with the limitations we posed, we decided to focus on the following three goals. First, do a literary research of AR together with Vuforia 5, to better know what is possible to do, and also how to do this. Second, to keep our research from becoming merely a literary study we wanted to create an application for Android that displayed some of the AR features. Third we decided to provide ideas for further development, as an inspiration for other projects within the same area of research

8 3. Workflow and implementation We started by using a library, called NyARToolkit, and managed to successfully implement this on a PC. Though easy to implement, NyARToolkit didn t contain all the functionalities we needed, neither was it especially suitable for a mobile platform, and we switched to Vuforia. The library we have been using, Vuforia, is a library developed by Qualcomm, for AR and smartphones. This library contains a lot of resources, along with a couple of tutorials which makes for quick learning. After a period of researching Vuforia, setting up the hard and software, we started by successfully implementing all of the tutorials and making sure we understood the code. Next step was to create our own application. Since we decided to create an application for Android, we used Vuforia together with the Android SDK and Eclipse. We wanted to do something that involved both Uppsala University and Schlumberger, so we decided to try to do something with their logotypes. To do this we had to first make the application be able to recognize the logotypes. This is currently being done using a web service that Qualcomm is providing. It works by uploading an image of the marker you want to find, let their classifier find characteristics in the image, and finally it presents the marker together with a grade of how easy the marker will be to spot. For example our markers for Uppsala University and Schlumberger were not that good. This depends on the markers not using the entire image space; there are a lot of gaps of plain white which doesn t help the image recognition. More so, the logotype of Uppsala University is almost perfectly rotational symmetric, this is making it difficult for the algorithm to find out what is up and what is down. As a last problem, both images don t have sharp edges, or differ greatly in contrast, which is also something that the image analysis algorithm is looking for. When done you can download the markers as two files and attach them to you application. You get an.xml file containing the names and sizes of the markers and a.dat file which is containing the data. 8

9 Figure 3: The Eclipse development environment. As soon as the application can recognize the marker it s time to decide what you want it to do when it finds the marker. We wanted to produce something that was showing the possibilities of AR, and that also was linked to the project and the companies. Therefore we decided to project text strings and one rotating logotype, all done in 3D. The creation of text is a bit tricky; this is because OpenGL, which is used in Android, doesn t support easy dynamical text creation and projection. It is possible to display text on the screen relatively easy, but since we wanted the text to be produced on the marker, we had to do find a way to do this. We ended up having to create the text in Blender 6, an open source 3D content creation suite. This meant that the text ended up being in 3D, but at the cost of it taking much more time than just writing a string like one would do for a normal JAVA or C++ program. The logotypes were also created using Blender, together with Inkscape 7, a program that can automatically trace an image and turn the image into scalable vector graphics in the.svg format

10 Figure 4: Rendering the Uppsala University logotype in Blender. Finally to import the objects into our application, we had to use a Perl 8 script that turned the.obj file into OpenGL graphics. The objects don t contain any textures, simply for the reason that they were already large, and were content because we could apply generic textures as long as we accepted that our objects could only have one color. To display the objects as we wanted them to be displayed, we had to rewrite and write a lot of new code. There are a lot of smart functions inside the Vuforia library though, and this has helped us a lot. We decided to let the logotypes spin, and be displayed above the marker, while the text should be static and displayed in the marker s plane all of which would help to show the possibility of walking around the object since it is locked to the marker and not to the screen. To make the object spin we created a function that uses the difference in time in between two calls for the system time, and rotating the object before projecting it. Doing this often makes it look like the object is spinning

11 We created one more app as well, to show the possibilities of virtual buttons. Virtual buttons are a function that creates buttons on the marker that when pressed sends a signal to the application to react. In our example, we created a spaceship that can rotate and move forward and backward. This also makes for a nice AR effect, where if you look at the spaceship right on, and push the button to make it fly away, it goes out of screen quickly. But, if you move the screen so that you see the space ship from behind, you can see it fly all the way out to periphery and disappear. 11

12 4. Results and conclusions Two different applications for Android are implemented. One shows text and an augmented 3D-logotype of Uppsala University or Schlumberger when the app detects their respective logotype. Logotype app This application has five different markers. Three of the markers are different versions of the Uppsala University logotype and two markers are different versions of the Schlumberger logotype. Depending on whether the Uppsala University logotype or the Schlumberger logotype has been found it does different but similar things. If the Uppsala University logotype (in any of the three different versions) has been detected, a red 3D augmented logotype is shown spinning above the marker. The position of this augmented logotype will be in the middle of the found marker but a small bit above the plane of the marker. In addition two pieces of text that lies in the plane of the marker will also be visible. These text pieces are 3D-objects but since it is placed in the plane of the marker it will give the appearance of being 2D. A third feature is also that when the Uppsala University logotype has been detected an options-menu button is changed from linking to google.com to instead linking to uu.se. When the Schlumberger logotype has been detected, the red augmented Uppsala University logotype is changed to a spinning Schlumberger logotype and the link is instead to Schlumbergers homepage slb.com. Figure 5: Our logotype app displayed on an Android device. 12

13 Virtual buttons app This app uses a generic image as a marker. This image consists of a wood cabinet combined with four different buttons. The main feature of this image is the four buttons with which the user can interact. The Image itself is taken from the Qualcomm examples and is used because of its good properties as a marker. When the marker has been found by the app a spaceship will be drawn on the screen above the plane of the marker. The buttons are best described as virtual buttons and can t be interacted with on the screen of the smartphone, but must instead be interacted with as if they existed for real on the image. To press and interact with the buttons the user must cover the button that he wants to interact with from the view of the smartphone camera. The buttons have different uses and you can accomplish four different things by pressing the different buttons. The leftmost button makes the spaceship roll right in its flight path. The second button makes the spaceship roll left and the third and fourth button makes the spaceship travel either forward or backward. The buttons works as press and holdbuttons meaning that when the first button is pressed, the spaceship will start rolling right with a constant speed. When the button is released the spaceship will stop rolling and maintain the angle at which it stopped. The same approach is used for all of the other buttons as well. Figure 6:A screenshot from an Android device using the virtual buttons app. Conclusions We think that we have succeeded in reaching our goals, by researching AR and creating two applications. On the way we managed to present a couple of possible uses for AR, that in our meaning, points in the direction that AR is a medium that is worth developing and can be of good use. 13

14 5. Discussion Difficulties The main difficulties within our project were first and foremost to decide what we were going to focus on. AR is such a big area, with a lot of possibilities. The project course plan practically only stated that we should do something within AR. Further on, we had to do some research before we found Vuforia, and we also tried with another library before Vuforia. We weren t really content with the functionalities and the demands of NyARToolkit and therefore we came to the conclusion that we might have to do some more research to find a better suited library. We also had some problems due to the fact that almost all new computers nowadays runs on 64-bit systems, and the first library that we used (NyARToolkit) used some very old components that demanded a 32-bit system. This was however quickly solved when we found Vuforia, since it is updated and supports almost all platforms. The biggest challenge perhaps was that we had to learn how to create objects in 3D, and to convert these into OpenGL standard. This is due to the lack of easy support for writing text within Vuforia and OpenGL. There are a few workarounds but due to the limitations in time for this project we decided to do this the same way it was presented in one of the tutorials we found. To do this we had to learn how to create 3D objects, and we chose a program called Blender. This was completely new to us and therefore took some time to get a hold of. Possible developments of our apps As a part of this project, we also decided that we would present a couple of possible improvements, so that further developing could be done by new groups. Since the project course only is 15 credits, this means that no group will be able to do all of the proposed tasks, but our propositions should instead be considered as inspiration to new project works. First of all we would like to see improvement being made in the image analysis. This is a part where we haven t done anything and as a result we ended up relying solely on the built-in image analysis in Vuforia. Though it works fine, it has some restrictions that keep us from using whichever picture we would like, and this is something we see as a potentially big improvement. The creation of markers is the second area we have thought of. Right now, markers are being made using the Vuforia homepage, by uploading the desired image, and then downloading one.dat and one.xml file. We would like to see the creation of markers being so simple that the user should be able to take a photo with a camera, and then choose to use it as a marker. This together with the improved image analysis could make it possible for any marker at all to be used. The third proposition is to work on simplifying the creation and displaying of text. This should be one of the simpler tasks, and make a big difference. In the end a lot of information is written, thus you can easily imagine that text will be the most projected object. If it would be possible to create the information dynamically, one could also have 14

15 updated information being projected. Together with internet access it would be possible to use updated information from homepages, process these and display the info. One can also use the example above and add, to the newly taken photo of a marker, a couple of sentences describing the marker or maybe even a link to the homepage where the information should be gathered. The probably biggest and most important improvement however might be to be able to use some kind of cloud-based storage system. If all the markers and information can be kept online, it would dramatically decrease the size of the application, and in the meantime making it possible to store almost unlimited amount of data. Again using the previous example, one can think of a database that isn t local anymore, but expanded by users worldwide. This would also make for a database that expands very quickly and at almost no cost at all. The application has to be connected to the internet at all times, but together with some smart search algorithm, should be quick in finding if the current picture is a marker, and if so what information is connected to this marker. The search algorithm also has to keep a check so that two users don t upload two versions of the same image and by doing so making it hard for the image analysis to detect which of the pictures is most similar to the camera feed. Possibilities for Augmented Reality We can see a lot of areas where AR could be used, some of them now and some in the future. Video games is a market that is constantly developing and pushing its frontiers forwards toward more realistic gameplay and new and spectacular gaming experience. With that in mind, we think that AR could be a huge success in certain video games. With the use of an arena and gaming glasses, one could create football, war or adventure games. You would need to keep track of the players, their movements and actions, but since this is already accomplished using for example the Kinect 9 tool for Xbox 360; this shouldn t be a main concern. To be able to offer a Battlefield or Call of Duty 10 experience where you re playing using your body to move, having a 360 degree of sight and the feeling of being inside the game, would dramatically improve the experience. One can also think of more practical possibilities, such as furniture fitting. Here you can imagine persons that are thinking of buying a shelf; they want to know whether or not it fits in their home by size, color and style. Produce a simple marker and put it where you want the shelf, and by using your mobile device you can see the actual size and style of the object, and of course change the color all by using AR. AR could also be used as installation walkthrough, whether we re talking furniture assembling with IKEA products or for example electronic devices. Recently an American show produced an example of how to use AR as a tool to install a router, with the mobile device recognizing the type of router and also which cables should be inserted in which Battlefield and Call of Duty are two popular video games, in the FPS (First Person Shooter) genre. They exist in numerous editions but all of them are in a war setting. 15

16 order and position. 11 Another idea is to use AR as a presentation tool. This can for example be used for companies who want to demonstrate their latest product, and to make it possible for potential buyers to see the actual product in 3D and be able to move around it using a more physical approach than just watch the product on a computer screen. It can also be thought of as accompanying other information, and extending its possibilities. Since May 2012 there is an app for iphone that together with the Metro newspaper lets the user see video clips of news that the paper otherwise would be restricted to describe in text and maybe publish a picture of. 12 Since most papers already feature a web version where they have video contents, this is a simple and cheap way to offer this functionality to customers who prefer to read a physical newspaper but still are interested in additional material. The best example of how highly rated AR is today is the looming introduction of Google glasses. That such a big company has invested a lot of resources into an AR project indicates that they really have confidence in the technique. Google glasses are believed to have a huge impact on the way and speed at which we will retrieve information. Without the obligation of using your hands, they can let you find out which bus that will take you home, who built the building across the street or what groceries is on sale in the local shop and which recipes that goes hand in hand with these. 11 Mills, Matt;

17 6. Acknowledgements We would like to thank Maya Neytcheva for her help and support with our project. She gave us valuable information and set up our office were we worked on this project. We would also like to thank Sergio Courtade, and Schlumberger, who came up with the idea to the project. It s been a really interesting and challenging project, and we have really enjoyed working together with Schlumberger. At last we also would like to thank Data-Mats for setting us up with computers, other hardware and the software that we needed. Thank you all! 17

18 7. References Kangdon, Lee. Augmented Reality in Education and Training. Tech Trends 56, no. 2 (March 1, 2012): doi: /s Mills, Matt. Image Recognition That Triggers Augmented Reality Video on TED.com, July y.html. Android. Accessed January 8, Blender.org - Home. Accessed January 8, Eclipse - The Eclipse Foundation Open Source Community Website. Accessed January 8, Get Started with Kinect - Xbox.com. Accessed January 8, US/Kinect/GetStarted. I Dag Får Din Metro-tidning Liv Metro, May 29, Inkscape. Draw Freely. Accessed January 8, NyARToolkit Project The NyARToolkit Project Website. Accessed January 8, The Perl Programming Language - Accessed January 8, Vuforia Augmented Reality Unleashed. Accessed January 8,