Augmented Reality System

Size: px

Start display at page:

Download "Augmented Reality System"

Agatha Daniels
5 years ago
Views:

1 Augmented Reality System ECE Senior Design Spring 2015 Final Report Team 1510 ECE Project Members: Daniel Adamowicz Amy Clark Victoria Tran Faculty Advisor John Chandy Office: ITE 437 Phone: (860) Graduate Student Advisor: Rajeev Ghimire Sponsor: Qualtech Systems, Inc. Professor Krishna Pattipati UConn ECE Department

2 Summary: The intent of our project is to create a product that will allow users to efficiently diagnose and troubleshoot a system through the use of smart glasses. The smart glasses will act as a voicecontrolled user interface for the mobile application PackNGo. The smart glasses will access Qualtech System s (QSI) remote diagnostic server (RDS) to retrieve necessary diagnostic and repair instructions and relay the information back to the user through the smart glasses. Background: To diagnose a given system, a technician needs to be familiar with that system. However, any one technician, no matter how experienced, can not possibly know all the causal relationships in a system or a family of systems. The problem becomes more acute over time as newer systems are developed, older systems become less common, and technicians ultimately have more things to remember. This knowledge gap could be potentially fixed with the use of diagnostic support systems, whose recommendations can be displayed on devices such as phones, tablets, or computers. This causes another problem: use of these devices requires the worker to look away and move their hands from the system, which can negatively impact their efficiency and safety. With the use of smart glass technology and Augmented Reality (AR), our group will use previously developed software to aid workers in diagnostic repair of a wide variety of systems. The project will aim to improve the accuracy and reduce the chance of worker error due to inexperience or the need to work with unfamiliar systems. The diagnostic information will be stored in an XML file and uploaded to the glasses. The application must retrieve information from the file and display it on the glasses. The main benefits of smart glasses over a mobile

3 phone are heads-up display of critical information and a completely hands-free experience by utilizing voice commands. Solution: Software: TEAMS-Remote Diagnostic Server (RDS) is a network-based application that gathers and processes data taken from active equipment and systems. The resulting diagnostics and system health information can be accessed via web browser. All results (e.g. procedures, diagnosis, testing) are recorded to TEAMS-RDS for later analysis and future reference. PackNGo is a software product that is part of QSI s remote solutions. It extends the Guided Troubleshooting functionality from the TEAMS system to smartphones and tablets as a mobile application. The application utilizes JavaScript and communicates with TEAMS-RDS. When PackNGo is opened, the user inputs the system being diagnosed and selects any relevant error codes and available tools from a checklist provided by PackNGo. PackNGo then displays a series of instructional checks through multiple pages. The user answers yes or no questions to obtain the most likely fault causing the problem in the system. One aspect of our design is to take the current UI as shown in Fig. 1 and adapt it to create an efficient display for smart glass. We plan to utilize this current software and adjust it to be controlled with voice commands to move through the troubleshooting questions and to bring the user to a repair solution. The final product should have no remaining touch controls for navigation. Android Studio is our main source of creating our Glass application. It allows us to create a fully functional UI and process the information parsing using the Java programming language. Android Studio is an ideal way to work with a Gradle based system which is what is used for google glass. Android studio displays the project files in a way that is easy to work with. Each of the classes which are defined as activities are organized and easy to navigate. The project view in android studio groups the files based on

function. For example it groups the build files, manifest files and resource files into there own folders so it is easier for us to manage our project.

In addition we also used Eclipse to create the parser and check that it was working correctly.

4 function. For example it groups the build files, manifest files and resource files into there own folders so it is easier for us to manage our project. Android studio was the main resource for programming google glass. In addition we also used Eclipse to create the parser and check that it was working correctly. Eclipse can use console commands to output the information and check that it is correct. This is easier and faster to do than on Android studio. Eclipse and Android studio have the same libraries that we can use so there is no problem importing the code to Android studio later. Current UI on mobile device

5 Fig. 1. Screenshots of current version of PackNGo. The top two screenshots are the lists of user inputs to customize PackNGo s instructions. The bottom two screenshots are examples of PackNGo s yes/no interface and sample instructions.

6 Use-Case Diagram Fig. 2 Use-Case Diagram of network integration of final product

7 Sequence Diagram Fig. 3: Sequence diagram, showing ideal flow of interaction of all of the systems in final design Hardware: Initially, our team researched technical specifications to narrow down our choice of smart glasses to two brands: the Epson BT-200 and Google Glass. We considered attributes such as display size, processing power, battery life, and wifi/bluetooth connectivity. It was also a requirement that the device have a microphone in order for us to incorporate voice commands. The Epson BT-200 is a simple smart glasses model we wish to use to integrate PackNGo

8 with as a proof of concept. The BT-200 is operated by a wired trackpad in the user s hand which operates similarly to a touchpad mouse on a laptop computer. The glasses use a 960x540 pixel display that appears to float in the center of the user s vision. The main screen is similar to a desktop on a computer: there are icons for different applications and interaction is done using a mouse pointer controlled by the touchpad. The relevant technical specs of the BT-200 are 1 GB of memory, 8 GB of storage space, and Android operating system. [1] The battery life for the Epson is advertised as approximately six hours at room temperature, although our team found it to be closer to 3-4 hours in testing. Due to the BT-200 s similarity to a normal computer, the UI changes necessary to integrate the existing PackNGo software is minimal. The BT-200 is considered only a proof of concept as voice commands are not officially supported and would require signal processing analysis to fulfill the hands-free requirement of the design. We have selected Google Glass to be the brand of smart glasses for our final design solution. Glass supports voice commands, but its general user interface is significantly different than the BT-200 or a computer. The screen for the glasses is a prism positioned in the top right corner of the right lens. This leaves the rest of the user s field of vision unobstructed. Google Glass has a minimalist style UI that has much less space for text and pictures than a computer screen. There is also no support for direct mouse clicks or touch control. This means the current PackNGo design would not work at all on Glass and must be completely redesigned. Glass operates almost entirely on voice commands by default, and applications written in the Google Developer Kit (GDK) can easily integrate voice commands. The relevant technical specifications for Google Glass are 1 GB of memory, 12 GB of storage, and Android 4.4 operating system. [2] Critics report the battery life of the Glass to be about three to five hours of extensive use [3] which we have found to be approximately correct.

9 Theory: For the Epson model, the PackNGo functionality is essentially mirrored on the BT-200 interface as the BT-200 is usable similarly to a computer. PackNGo required no additional modification to function on the BT-200. For the Glass model, the application must have access to an XML document containing information for some given system. This XML file is generated by Qualtech software and given to us directly. The Android application was created from scratch to display the necessary information in an organized manner. Part of this application parses the XML file and extracts the necessary information, then displays this information on the Glass. The application is opened with and fully controlled by voice. After development of this application, we will test and verify the compatibility of our application with both models of smart glasses. The application running on the Glass will then be tested in a field environment. The tree diagram can be interpreted as in Fig. 4:

10 Fig. 4: Visualization of tree diagram structure We will need to understand the diagnostic process of PackNGo to ensure that we can utilize QSI data through the XML file. Contact with QSI chief architect and software engineer was necessary to avoid complications. We will need to export the current TEAMS model of air handling unit (AHU) to RDS to the QSI server to convert diagnostic information to XML file format. This system will be the test environment for the final product. The final product should function properly in environments such as automotive facilities, aerospace industry, hospitals, semiconductor fabrication plants, and more, all of which are potential QSI clients.

11 Preliminary Observations: After some basic testing of the Epson BT-200, we have noticed that the design is not ideal for a hands free program. First, the touchpad input is connected by a wire. This feature is, unfortunately, bulky. Ultimately, this means that the BT-200 is not intended to be a hands-free device without significant manipulation of sensor data within an application. In order to utilize voice commands on the Epson, third party speech recognition software would need to be integrated into the QSI software. However, we can still use this device to display PackNGo as is for a proof of concept for future augmented reality projects. Another issue is in the way the BT-200 screen is displayed. The screen appears to float in the center of the user s field of vision. Although the opacity can be adjusted, it is still possibly disruptive for someone trying to work on a system. For future augmented reality projects, this feature could potentially be useful for projecting useful images and data directly onto a system. For the purposes of this project, the positioning of the screen is only a disadvantage. The biggest issue with Google Glass is its heat dissipation. We found that using the Glass for extended periods of time (longer than 30 minutes) could cause the Glass to get excessively hot. A warning screen appears when reaching critical temperatures. We have experienced rare system crashes possibly caused by overheating. To remedy this, we recommend not operating Glass for longer than intervals of thirty minutes, and turning it off in between intervals to cool. The system could likely operate for longer, but in the interest of being safe, we recommend limiting its on time to thirty minutes. Glass voice commands can be overly sensitive for short, simple phrases. When the microphone detects human voice, it starts attempting to match its microphone data to all possible

12 voice commands. This means that a voice command could register even if it is said improperly or incompletely. For example, the command Take a picture sometimes works just by the user saying Take a pic or similar incomplete commands. The purpose of this is so that people with accents or speech impediments can still use the Glass. This causes issues with multiple voice commands that sound similar and are ready at the same time. A related issue is that once a voice command is processed (even if the user is not done saying it), the Glass reacts and, depending on the action, will potentially ready-up another set of voice commands. We encountered the aforementioned problems when using the contextual commands inside the application. To open the contextual commands, the user must say ok glass (this command can not be changed). After the ok glass command is processed, the program then waits for the next set of commands, which includes the command Yes, to continue traversing the PackNGo system data. We have noticed that: 1) ok glass seems to trigger before the phrase is finished, 2) prepares to listen for Yes, 3) hears glass (or at least, part of the word), 4) and then triggers Yes before the user has a chance to provide meaningful input. This inconsistency in our application could be fixed by changing the voice commands from Yes and No to something else; unfortunately, there does not seem to be any logical replacement for these words that would make sense for any and all given diagnostic prompts. This is simply because the diagnostic prompts are written in a way so that all PackNGo buttons are the same for every step (yes and no).

13 Specifications: Product Google Glass Epson BT-200 Price ($) $ $ Display Prism projector, pixels pixels screen Onboard Storage (GB) 16 flash total (12 GB of usable memory) 8 GB OS Android 4.42 Android Power Lithium polymer battery (2.1 Wh) Lithium poly 2720mAh CPU OMAP 4430 SoC dual-core TI OMAP GHz dual core Memory 1GB RAM (682MB available to developers) 1 GB Sound Bone conduction transducer headset, surround sound Camera Photos 5 MP, videos 720p VGA Connectivity Wi-Fi b/g, Bluetooth, micro USB Wi-Fi (IEEE802.11/b/g/n), bluetooth, microusb 2.0 Table 1 - Relevant technical information for both brands of smart glasses Implementation: The XML parser was written in Java and built using Eclipse. The parser utilized the DOM and SAX existing parser classes to gather data from the XML. The data we need to

14 retrieve from the XML is stored in nodes with numbered labels. The data we need to extract from each node is the label number, instruction string, outcome actions (typically yes/no), and the outcome pointers. The label number keeps track of which step the program is currently reading. The instruction string and outcome actions tell the program what to print to the screen. The outcome pointers tell the program which node to go to next based on what command (yes/no) the user says. The parser simply scans the XML file for the relevant tag names and returns the necessary data. Glass applications can be designed using one of three templates: static cards, live cards, or immersion. We chose to create an immersion application because it gives us maximum control over the Glass interface. Immersion applications allow us to continually update information on the screen without a heavy impact on Glass s processing power. Due to Glass s minimalist UI, the amount of information displayed on each screen is limited. The yes/no buttons have been changed to voice commands, so they do not take any real estate on the screen. The pictures and text can be displayed well by resizing them slightly. Fig. 5: Example of application screen with instruction string.

15 Fig. 6: Contextual pop-up menu containing typical yes/no choices. There was one major issue with our original implementation. The XML file to be parsed was included in the Java package and uploaded to Glass. When the parser runs, the program can not locate the XML file and is completely unable to retrieve any data. As of this time, the reason for this problem is unknown. Using Windows Explorer, very limited file paths can be accessed, so it is difficult or impossible to locate files uploaded to Glass. As a workaround, we converted the entire XML file to a string and wrote it explicitly inside the parser code. Doing so requires the XML to be modified to eliminate all quotation marks (due to Java string syntax), done using a find-and-replace tool in any text editing program. This string workaround was able to be parsed properly by our application [4]. Budget: The main focus of this project is developing an existing software for use on Glass. The Epson BT-200 and Google Glass are both provided by the graduate student group and cost our

team nothing. The Integrated Development Environments (IDEs) and all Qualtech software is free for us to use. Since the entire project is software based, no other parts need to be purchased.

16 team nothing. The Integrated Development Environments (IDEs) and all Qualtech software is free for us to use. Since the entire project is software based, no other parts need to be purchased. Therefore, the budget for this project is zero. Timeline: The main focus of the first semester was to research the many types of different smart glasses available and choose one most appropriate for the task of aiding remote diagnostics. We also worked on Android development and gaining access to the current PackNGo software, TEAMS-RDS. We experimented with BT-200 features and ran the original PackNGo software on them. In the second semester, we began full development of the Glass application. A mock-up UI for the Glass application was designed. We analyzed the XML file and developed the Java parser. We developed the full Android application with integrated voice commands, and ran the parser program on the Glass to extract data.

Appendix: Citations: [1] http://www.epson.com/cgi-bin/store/jsp/product.do?sku=v11h560020 [2] https://support.google.com/glass/answer/3064128?hl=en [3] http://www.techradar.

17 Appendix: Citations: [1] [2] [3] Software References: User Guide: 1) Create an XML file for a system using TEAMS software. 2) The XML file to be used must be modified (refer to Implementation, [4], page 14) and written directly into the string variable in the Java package. 3) To upload the application, use Android Debug Bridge (ADB) Sideloading. For more

18 information refer to 4) Once the application is uploaded, open from the main menu with Ok glass > Check system 5) The first step of the used XML file will automatically appear. To continue, say Ok glass followed by the related voice commands (e,g, Yes or No ) 6) To go back to the previous step, say Ok glass followed by Go back 7) To exit the application, swipe down on the Glass touchpad. Acknowledgements: We wish to acknowledge and thank the previous senior design teams and graduate students for providing real-time monitoring and creating a link between the sensor values of the HVAC system in the Information Technology Engineering Building (ITEB) at UConn, to the failure protection software, TEAMS, run by QSI. We also wish to thank Qualtech Systems, Inc. and the University of Connecticut Electrical Engineering Department, for support with funding, advising, and sponsoring this project. Special thanks to our graduate advisor Rajeev Ghimire and project advisor John Chandy for guiding us through development.

Smart Glasses & Applications. Presenter: Xu Qiu! Advisor: C.-C. (Jay) Kuo!

Smart Glasses & Applications. Presenter: Xu Qiu! Advisor: C.-C. (Jay) Kuo! Smart Glasses & Applications Presenter: Xu Qiu Advisor: C.-C. (Jay) Kuo 1 1 Outline Introduction Comparison of Smart Glasses Apps for Developing Platform Development and Implementation of Computer Vision