AugLAC - Generic Framework for Augmented Location Aware Commerce

AugLAC - Generic Framework for Augmented Location Aware Commerce E.M.P. Gunatunge, H.Y.M. Hewagama, S.G.S. Hewawalpita, I. Perera, Y.B.N. Udara Dept. of Computer Science & Engineering University of Moratuwa. Moratuwa, Sri Lanka. {mihiri.10,yahani.10,supungs.10,indika,udara.10}@cse.mrt.ac.lk Abstract In this paper we present AugLAC, a framework which integrates common features of location aware services, e- commerce and augmented reality into a single infrastructure for the development of location aware mobile applications. The research aim is to generalize the proposed framework to suit different business domains and to allow rapid development of applications using a component based architecture. Although this framework is mainly targeted for retail business domain it can be effectively used for other domains and use case scenarios where the basic principles apply. Keywords location aware services, e-commerce enhancements, augmented reality, mobile apps. I. INTRODUCTION With the advancement of technology of smart devices people began to highly dependent of those for various kinds of aspects. Location finding is one of most important aspect among them. Location awareness has been growing in importance in providing services for mobile devices and many business firms are looking to use this technology to enrich their customer services and widen their markets. On the other hand most smart device users are attracted to 3D effects and visualization. This has made how the solutions are presented to the end user a major concern of the solution providers. Augmented reality is a new trend of presenting mobile solutions that made possible with the wide use of smart phones and the significant increase of their processing power. It gives more realistic and intuitive feeling since it integrates the real world environment and virtual environment together. A combination of location awareness and augmented reality can be used together to provide innovative solutions for various business problems and we named this domain as augmented location aware commerce. Several frameworks have been implemented specifically to develop general mobile AR applications. The most popular framework ARToolKit [1] focuses more on tracking technologies and other frameworks such as Studierstube ES [2] focus software development aspects by using reusable component based design. But to use them for augmented commerce applications, e-commerce features need to be developed from scratch. PromoPad [3] is an in-store e-commerce system, which provides shopping assistance and personalized advertising through augmented reality. Pocket BargainFinder [4] is another similar system that can scan items and find bargaining prices on Internet. These systems are domain specific and do not feature a generic platform to develop mobile augmented reality applications for various domains. Many applications are currently available in the area of location aware services using GPS technologies. Such applications are capable of providing location information in outdoor environments where the range of precision of the location is quite large. GPS based solutions are not adequate for indoor environment since the location information have to be more precise than in the outdoor environments. Many researches has been done on indoor positioning systems and various techniques including marker based [5], Bluetooth based [6] and Wi-Fi based methods have been introduced. In this paper we present AugLAC, a framework that integrates location aware services, augmented reality and some common e-commerce features that can be used rapidly develop applications in the domain of augmented location aware commerce. The idea is to provide a generic component based architecture with pluggable modules so it can be easily adapted to any business scenario. A component-based architecture which is built upon layers of technology is proved to be effective in rapidly developing applications are highly flexible to changes. As indicated by Martin Bichler, Arie Segev and J. Leon Zhao [7] Component-based e-commerce technology is a recent trend towards resolving the e-commerce challenge at both system and application levels. AugLAC framework also supports multimodal interactions which are essential to enrich the user interactions with augmented reality. The paper is organized as follows: First we provide an overview of the framework and its major components. It is followed by a section about the implementation of the framework, algorithms and mathematical models with technical details. Then we discuss the future work that need to be carried out to complete the framework.. II. OVERVIEW AugLAC consists of three major features; location awareness, e-commerce and business services and augmented reality with multimodal interactions. The main purpose of this framework is to allow developing rich applications rapidly with these features. For that the client side of the framework is developed in a component-based architecture, so features can be added or removed as components according to requirements of the application. 978-1-4799-4598-6/14/$31.00 2014 IEEE

Fig. 1.Client side architecture of AugLAC framework Some components such as augmented reality, location aware features depend on the positioning system and other feature such as multimodal interactions are independent components. A. Positioning System Positioning system addresses the location awareness aspect of this framework. Since our target domain is product based businesses, our main concern is on indoor positioning. There are many technologies available to choose between when designing an indoor positioning system such as marker-based, Bluetooth and Wi-Fi. Existing solutions for indoor positioning is limited to one of the above technologies. If a client is to choose such solution for his business, he is forced to use the technology that is supported by the solution. As a generic framework, our framework combines all the technologies mentioned above producing a hybrid model giving the client the freedom to choose whatever technology that is affordable, fitting and profitable to his business. Also this hybrid approach can provide more accurate results when several technologies are combined. For example, if both Bluetooth and marker based positioning are used in an application, positioning system will be more accurate and it can switch between the methods depending on the location and services as required. B. Business Services Components Business Services component associated with this framework can be divided into two sections as common business services and e-commerce related services. These include server side services as well as the client side services for application users within these two categories. Some example business services that this framework would provide are item catalog management which includes managing own item databases or retrieving data from existing databases, ability to search, gathering user statistics and advertising which can be used to display customized advertisements. Each of these business service would be provided by separate modules and this framework provide an extendible approach where particular business services could be added in to or removed from the applications based on the domain and the commercial requirements. C. Multimodal Interaction Components Traditional interaction methods are not sufficient to provide the expected user experience for ubiquitous mobile applications developed with AugLAC framework. So we expect to provide alternative methods of interactions rather than conventional windows, icons, menus and pointers (WIMP) interfaces. It will allow the user and the application to choose interaction modalities according to the user preferences, the task at hand and the various social and physical contexts of use. These components are the building blocks for designing user interfaces and interactions with this framework. It provides basic support for gesture recognition and audio interactions and includes strategies to switch mode of interactions. D. Dynamic Learning Module This module adds artificial learning capabilities to the framework. One such capability is learning the characteristics of the application environment to suggest best routes to navigate in the environment. This component has a combination of data mining and navigation mesh algorithms to provide suggestions based on the choices of the users. Another capability is to provide item suggestions based on the users previous choices. This module adds the framework the ability to identify each user s usage behaviors and patterns so that it can provide custom tailored services and advertisements to specifically suit the user. Server side of the framework is developed according to the architecture given in Fig. 2. The client-server interactions are handled through a uniform interfacee provided by a REST API where the server databases are accessed by client via a data service. The server holds two types of databases, AugLAC database where the data relevant for the framework functionalities are held and third party databases where the existing data about the application domain is held. Data that is extracted by the server modules from the databases are delivered as the client interacts through the REST API. Fig. 2.Server side architecture of AugLAC framework

III. IMPLEMENTATION The major features of the framework were broken down into separate modules and developed independently according to the component based architecture of the project. Modules are implemented to support multiple platforms, techniques and business models. The indoor positioning techniques are bundled together as a single module with interfaces to perform the tasks involving indoor navigation and tracking user position and orientation. This module includes few indoor positioning options including Bluetooth and Wi-Fi. The algorithms and tools that are used are described in details in later topics. Augmented Reality component is another module which handles the 3D visualization and displaying information layer. 3D visualization is a combination of marker based tracking and augmented reality while the information layer uses the Bluetooth or Wi-Fi tracking along with augmented reality display techniques. For the implementation of the framework we used two third party open source libraries. OpenCV [8] was used for image processing and camera mapping functionalities while Min3d was used for 3d rendering. One major problem with using OpenCV was due to its large size the framework became bulky though we needed only a small amount of its features. So we use a modified lightweight version of OpenCV in which we have removed unnecessary parts. Min3d s light weight was the main reason we chose it as the 3d rendering engine and it was also very simple and flexible to use. A. Bluetooth and Wi-Fi Based Positioning For the implementation of the Bluetooth positioning system we use NRF51822 2.4GHz Bluetooth modules as beacons which continuously transmits a pre-programmed identification number. This module has an inbuilt microprocessor and can operate without any external circuit with only power supply. It reduces the complexity of the beacons and power consumption making them operational without replacing batteries for a long time. Smartphone will get information from the surrounding beacons and responsible for localization itself by detecting the beacons. This also has an advantage since the device has no need to be in discovery mode so that it maintains the privacy. Since the calculation happen on device no data about the device is stored on servers. An initial calibration is required for beacons by giving an id and registering it in server along with their position details. Wi-Fi based positioning use the same approach, but instead of placed beacons it uses existing Wi-Fi access points. It is less accurate than the Bluetooth positioning module because this depends on the Wi-Fi signal strength which can vary from time to time. The relative locations of the known Wi-Fi access points within the area of application need to be calculated and stored in the back end database prior to the application deployment. For the location calculations, the dimensions of the building are mapped to an x, y, z co-ordinate system. Within the application, signal strengths of each access point is calculated in regular intervals and the position of the device i.e. the user position is calculated using that and locations of access points. In order to get a reasonable correctness, at least the locations and signal strengths of three Wi-Fi access points are required for location calculation algorithm. There can be errors due to change in signal levels, giving different position values even though the user stays idle. To minimize such errors due to rapid changes in signal levels over time, the position calculation algorithm uses the average signal level from several scans. Wi-Fi signal strength can also be affected by other factors, but such errors are inevitable and can only be minimized by increasing the number access points. As a future enhancement, a barrier detection algorithm can be introduced based on the user s current location, barrier locations and access point s location. B. Marker Based Augmented Reality Real time fiducial marker tracking is a standard mechanism in augmented reality which is simpler compared to feature tracking of natural objects. It is highly robust and works well under various conditions with a low computational resources. For the implementation of this module we used an existing square grid marker scheme. A marker is basically a black and white grid patterns surrounded by a black border which correspond to a five word bit pattern. Each marker contains 10 data bits and 15 redundant bits in a Hamming coding scheme for error correction allowing them to have 1024 different ids. In the marker detection process, we first apply an adaptive thresholding to detect edges in the image. Then contours are found in the resulting image and approximated as polygons to find polygons with 4 corners. Then each such polygon is projected to remove perspectiveness and internal pixels are scanned to identify valid markers. Using the Hamming coding scheme used in the marker, its rotation can also be detected. (a) (b) Fig. 3. Fiducial markers: (a) Marker scheme used in the system. (b) Example of marker detection in the system. This process is computationally expensive to perform on every frame of the camera feed, so as an optimization full image frames are processed only periodically. Once one or

more markers are detected, only the neighborhoods of the detected markers are scanned in the next frames until a full frame scan take place after a predefined number of frames. Then in the next step camera transformation and rotation matrices are calibrated from the known world coordinates and detected image coordinates of the identified markers. To augment virtual objects to the camera feed, a 3D scene is constructed in real world coordinates and its camera s position, orientation and roll are set according to the calibrated camera matrices. Then the 3D scene is rendered on top of camera feed resulting in a see-through like augmented reality display. C. Hybrid Positioning Although Marker based positioning is more accurate than Wi- Fi or Bluetooth based techniques it is limited in range and also it is not practical to place markers everywhere in the environment. As a solution we developed a hybrid technique to take the advantages of both ends. In this approach we use Wi-Fi or Bluetooth for default positioning and periodically scan the camera feed for markers. When a marker is detected we keep the last position returned by the Wi-Fi/Bluetooth module and shift to marker based positioning. Position updates from markers as long as they are in view and when they disappear from the camera range position get fixed at the last returned position from marker positioning module. When we detect a significant deviation from last known Wi-Fi/Bluetooth position we shift back to Wi-Fi/Bluetooth positioning. With this approach users can get continues updates location while they are on the move and also highly accurate positioning at specific locations with markers. IV. FUTURE WORK A. Server Components Implementing the server component is the most important thing that is to be done. The server component is vital to entire framework and can be considered as the heart of the framework. It is basically the backend of the framework where database related functions and business services are provided. The augmented application will highly depend on the services provided by the backend server. The functionality and the quality of service of the application will be determined by the back end server services where these services will be predefined in the server component. In order to register a particular user into an environment and to provide the data required to augmentation back end server will need to keep the user specific data, as well as data about the environment. Upon the request from the application, information will be retrieved from the server in this customizable manner. The server needs to provide secure services since some information may be specific to user. Some applications may have different groups of users where information of a particular group of users should be privately held whereas the other information can be publically shared. Therefore securing services where necessary is an essential attribute of the server. B. Augmented Advertizing The technology to pinpoint a mobile phone's location has a significant commercial value in the context of advertising. Location based advertising (LBA) is a new trend that exploit this and the advertiser can reach the customer when and where he is most likely to buy. We expect to provide advertising facility in the future within the augmented reality space in both push and pull approaches [9]. With the user behavior patterns, recommendations for the users will be provided based on their application domain. The advertising is to be further expanded into the areas of providing annotations targeting the customers of the domain. These will be customized based on the customer, location and the context. C. Motion Gesture and Sonic Interactions This will require some researches on its own and our primary focus is on motion gesture based interactions. With gesture interactions, the system should be able to identify predefined user gestures and respond accordingly to fulfill the user expectations. The objective is to provide an alternative method of interaction with applications when using the phone with a single hand and increased level of interactivity will make the application a better solution. The hand gestures recognition in real time [10] would be an added advantage where it increases the degree of interaction the user can have with system in real time. Although they seem fascinating, applying them in everyday life can be difficult as motion interactions should be usable in any ordinary situation and unintended motions should not trigger associated functionalities. There have been some researches on metaphorical natural gestures [11] and designing new unique gestures [12]. Sonic or audio based interactions are another alternative we expect explore which have been proven [13] to be very effective at improving interaction on mobile devices. Compared to motion gestures these are easy to implement and also has the potential to aid the visually impaired users as the main interaction model. D. Audio Augmented Reality Audio augmented reality is a less common but very effective aspect of augmented reality concept. Similar to augmenting visuals, sounds can be augmented to real environment. It can be used to provide audio narrations [14], sound alerts [15] or as an alternative interaction for visually impaired. One advantage is the users can emerge in audio augmented reality while keeping their hands free engaging in some other activity. V. CONCLUSION The latest trend in business/e-commerce solutions is closely related to introduction of various kinds of smart devices and number of business and e-commerce solutions have been developed targeting smart devices during the last decade. With this increased use of smartphones, there is a new opportunity to provide services to the customers based on their location. As a number of vendors are competing to promote their business solutions, the way the solution is presented to the customer, in other words, end user experience is very important. With

introduction of technologies such as x-box, new generation is attracted to interactive and visually high quality applications. Combining the augmented reality aspects into mobile applications is an attempt to provide the end user the similar kind of user experience. Even though there are business solutions that use location awareness and augmented reality with innovative ways, no attempt has been made to generalize the idea to match various business domains. The framework we discussed in this paper integrates common features of location aware services, e- commerce and augmented reality into a single infrastructure. It provides a set of tools to rapidly develop mobile business solutions with above features and multimodal interaction capabilities allowing the development of rich applications with great user experience. Also it is a highly generic framework and can be adapted to match various domains and requirements with a minimal configuration. Our solution is not limited to a single platform or a single device. It is robust and can be customized to any extent according to business requirements because of its component based architecture. Developing applications on top of the framework is very easy with simple interfaces it provides and guaranteed to provide best user experience. Indoor positioning and augmented reality features can be added to the applications according to the business needs since the framework supports not only one but few indoor positioning technologies and augmented reality techniques. Going further with the development of AugLAC, we expect to integrate visual search capabilities where the system will be able to scan the visual environment for a particular object or feature among other objects or features with respect to each application domain environment. Another such capability is gesture interactions where the system is capable of identifying predefined user gestures and respond accordingly to fulfill the user expectations. The objective is to provide an alternative method of interaction with applications when using the phone with a single hand. Finally we are focusing about audio augmented reality which is similar to visualizing augmented objects and framework will be able to augment audio based on location data when this capability is integrated. [7] M., Segev, A., & Zhao, J. L. Bichler, "Component-based e-commerce: assessment of current practices and future directions," ACM SIGMOD Record, pp. 7-14, 1998. [8] G. Bradski, "The opencv library," Doctor Dobbs Journal, vol. 25, no. 11, pp. 120-126, 2000. [9] G. C. Bruner and A.l Kumar, "Attitude toward location-based advertising," Journal of Interactive Advertising, vol. 7, no. 2, pp. 3-15, 2007. [10] W. H. Chun and T. Höllerer, "Real-time hand interaction for augmented reality on mobile phones," in Proceedings of the 2013 international conference on Intelligent user interfaces, 2013, pp. 307-314. [11] J. Ruiz, Y. Li, and E. Lank, "User-defined motion gestures for mobile interaction," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 197-206. [12] J. Ruiz and Y. Li, "DoubleFlip: a motion gesture delimiter for mobile interaction," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 2717-2720. [13] A. Pirhonen, S.A. Brewster, and C. Holguin, "Gestural and Audio Metaphors as a Means of Control for Mobile Devices," in Proceedings of ACM CHI 2002, Minneapolis, 2002, pp. 291-298. [14] B. B. Bederson, "Audio augmented reality: a prototype automated tour guide," in Conference companion on Human factors in computing systems, 1995, pp. 210-211. [15] N. Sawhney and C. Schmandt, "Nomadic Radio: Scaleable and Contextual Notification for Wearable Audio Messaging," in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 1999, pp. 96-103. REFERENCES [1] D. Wagner and D. Schmalstieg, "Artoolkit on the pocketpc platform," in IEEE International Augmented Reality Toolkit Workshop, 2003, pp. 14-15. [2] D. Wagner and D. Schmalstieg, "Experiences with handheld augmented reality," in Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium, 2007, pp. 3-18. [3] W. Zhu, C. B. Owen, H. Li, and J. H. Lee, "Personalized in-store e- commerce with the promopad: an augmented reality shopping assistant," Electronic Journal for E-commerce Tools and Applications, vol. 1, no. 3, pp. 1-19, 2004. [4] A. B. Brody and Edward J., "Pocket BargainFinder: a handheld device for augmented commerce," in Handheld and Ubiquitous Computing, Springer Berlin Heidelberg, 1999, pp. 44-51. [5] A. Mulloni, D. Wagner, I. Barakonyi, and D. Schmalstieg, "Indoor positioning and navigation with camera phones," Pervasive Computing, vol. 8, no. 2, pp. 22-31, April-June 2009. [6] K.C. Cheung, S.S. Intille, and K. Larson, "An Inexpensive Bluetooth- Based Indoor Positioning Hack," Massachusetts Institute of Technology, Cambridge, USA.,.