SmARt Shopping Project

Transcription

1 Detailed Design Report SmARt Shopping Project Sponsored by ASELSAN V2, 2010

2 Table of Contents 1. Introduction Problem Definition Purpose Scope Overview Definitions, Acronyms and Abbreviations References System Overview Design Considerations Design Assumptions, Dependencies and Constraints Assumptions and Dependencies Design Constraints Design Goals and Guidelines Reliability Usability Portability Extensibility Data Design Data Description Product Table Section Table Salesman Table Data Dictionary Entities Methods System Architecture

3 5.1. Architectural Design Description of Components DatabaseHandler InputHandler Processor OutputHandler Design Rationale User Interface Design Overview of User Interface Screen Images Screen Objects and Actions Detailed Design DBHandler Component Classification Definiton Responsibilities Contraints Composition Interactions Resources Processing Exports InputHandler Classification Definiton Responsibilities Contraints Composition Interactions

4 7.2.7 Resources Processing Exports Processor Component Classification Definiton Responsibilities Contraints Composition Interactions Resources Processing Exports OutputHandler Component Classification Definiton Responsibilities Constraints Composition Interactions Resources Processing Exports Libraries and Tools OpenCV Eclipse CDT MinGW MSYS Qt

5 8.6 MySQL OpenGL Time Planning Term 1 Gannt Chart Term 2 Gantt Chart Conclusion

6 1. Introduction This Software Design Document provides a complete description of SmARt Shopping Project sponsored by ASELSAN, as a final design project of Middle East Technical University Computer Engineering Department, including all design issues. This document will guarantee that the design will correctly implement all the functionalities identified in the SRS document, it will be easily understandable, efficient, and suitable to change Problem Definition In order to define the problem, one should imagine a scenario in which a standard customer enters to a supermarket. When a customer enters to a supermarket, he finds himself in a huge crowded environment that has lots of sections and messy amount of products. For a customer to achieve his purpose he should make a lot of effort to do other things at the same time. For example finding the exact places of the products he is searching for, while trying to carry the products that he is willing to buy and paying attention to his surroundings such as other people would be a burden for a standard customer. Our team intends to solve this problem which is making shopping easier for an ordinary customer by improving the interaction between the customer and his environment. In other words, the real world problem which this project aims to handle is to collect information from the surrounding area without any effort by a standard user, namely customer in a supermarket. The desired information to be collected is expected to consist of finding the locations of the desired products, and classifying the products according to their costs and different other categories in huge supermarkets. Moreover the information about the section and the salesman is expected to be achieved Purpose This Software Design Document provides the design details of SmARt Shopping Project in the scope of Middle East Technical University Computer Engineering Department Graduation Project. This document aims to represent a software system that is used as a medium for communicating software design information, namely for SmARt Shopping Project that implements augmented reality. This Software Design Document addresses the ones who are going to develop this 5

7 system, ASELSAN members and the departmental instructors who are responsible for graduation projects. It will give overall information about the desired project and necessary steps in order to accomplish the task, such as system overview, design considerations, data design, system architecture, user interface design and detailed design. The SDD is the primary reference for code development. As such, it must contain all the information required by a programmer to write code. Furthermore, the purpose of this document is to identify, contain and eliminate risks on a project. Also this document ensures that risks are identified and assessed as early as possible in the project, which allows the project manager to make informed decisions as to the direction of the project so that it may lead to a successful conclusion. This software design document is the how to of software life-cycle documentation. It details how the software requirements should be implemented Scope The project which is going to be presented in this document is called SmARt Shopping Application. This application is planned to be used by any person who is willing to do shopping in huge supermarkets. It is designed to run on a hand-held mobile device connected to a head-mounted display. It will allow customers to improve their quality of shopping, and make it easier for them. The promised functionalities of the application will be implemented by means of object recognition, object tracking and using augmented reality tools. By the application, the real world scene captured in the supermarkets will be enriched by the information obtained from the real-time inputs which allow customers to do an interactive shopping. The application will start running by the desire of the customer. First, the customer using this application is expected to choose from the products which are defined and listed previously in the application database. After the customer chooses the demands and lists them, the application interface will change into a real-time camera which is capable of recognizing the interest objects such as section labels including the desired products, products themselves, or employees of the market. The interact with the user will be maintained by labeling the moving or non-moving recognized interest objects in real-time. Since the user will use a head-mounted display, he will be able to combine the real world with the enriching data without any distraction. The application will also provide information about the found products cost and quality and popularity. The interaction will continue until the user quits application. 6

8 Labeling objects will be both according to the chosen list and the previously specified features of the identified objects. A section, which has products on sale, is also labelled when it's recognized even if the customer does not add any products to the list from that section. The sections that include the chosen products will be labelled different than the ones who has sale, which can be understood from the features of the sections. In order to maintain the accuracy of the application, some assumptions will be made such as tags that will help us to recognize the section labels, the shapes of the products, and some color and tag assumptions of products will be made in order to give information about cost or quality of the products Overview This Software Design Document consists of nine parts. First, in the introduction part, it defines the real world problem that our team intends to solve, then explains the purpose of this document. Immediately after that, the scope of the project is explained in an exhaustive way, the contents of the document is summarized, and important terms, acronyms and abbreviations relevant to the project is clarified. In the second part which is called system overview, a general description of the software system including its functionality and matters related to the overall system and its design is provided. The intended goals, objectives and benefits of our project is briefly explained. In the third part, the concern is design considerations. Special design issues which need to be addressed or resolved before attempting to devise a complete design solution are noted in this part. This part is categorized into two sections, first, design assumptions, dependencies and contraints are defined, and secondly design goals, guidelines, principles that our team embraces are covered. In the fourth part, data design is elaborated. The way that our team transforms information domain into data structures, the major data and system entities that are stored is explained. Subsequently, a data dictionary is provided in order to provide a detailed description of the system entities and major data, including data objects, their attributes and methods. The fifth part covers the system architecture issues. This document covers this topic in three parts. In the first part, namely architectural design part, the system structure that our team has chosen for SmARt Shopping application is developed. The modular structure of the application is defined and the relationships between the modules to achieve the complete functionality of the system is expressed. Relevant diagrams are provided. In the second part, the design is observed in subsystems, and the functional descriptions and object descriptions are provided. Relevant diagrams 7

9 such as Sequence Diagram, and Hierarchy Diagram are provided. In the third part, the reasons for the way of decomposing the system that our team has chosen is explained. The sixth part depicts the user interface design. The overview of user interface is given. Some screenshots showing the interface from the user s perspective are provided. Screen objects and actions are explained. In the seventh part, detailed design of the project is explained. Classification,definition, responsibilities, constraints, composition, interactions, resources, processing, exports issues are covered for every component. In the eighth part, libraries and tools that our project is dependent on are explained. In the ninth part, time planning of our team is given. The gantt charts for fall and spring terms are provided. The last part covers a brief conclusion of this Software Design Document Definitions, Acronyms and Abbreviations SDD: Software Detailed Design Report SRS : Software Requirements Specification AR: Augmented Reality, a term for a live direct or indirect view of a physical real world environment whose elements are augmented by virtual computer generated sensory input such as sound or graphics. OpenCV: Open Source Computer Vision Library Eclipse: Eclipse is a multi-language software development environment comprising an integrated development environment (IDE) and an extensible plug-in system. Our project uses Eclipse CDT for C/C++ projects. Eclipse CDT: The CDT provides a fully functional C and C++ Integrated Development Environment based on the Eclipse platform. IDE: Integrated Development Environment ER: Entity Relationship Diagram, a specialized graphic that illustrates the interrelationships between entities in a database. ER diagrams often use symbols to represent three different types of information. Boxes are commonly used to represent entities. Diamonds are normally used to represent relationships and ovals are used to represent attributes. GUI: A graphical user interface, a type of user interface that allows users to interact with programs in more ways than typing such as computers. 8

10 UML: Unified Modeling Language, a standardized general-purpose modeling language in the field of software engineering. Qt : A cross platform application and user interface framework that is widely used for developing application software with GUI (graphical user interface). UI: User Interface Cross-Platform: An application which can work in different Operating Systems. DLL: Dynamic-link library API: Application Programming Interface POSIX: Portable Operating System Interface for Unix DB: Database 1.6. References [1] [2] [3] [4] [5] [6] [7] IEEE Std IEEE Recommended Practice for Software Design Descriptions. Fowler, Martin. UML distilled: a brief guide to the standard object modeling language. Booch Jacobson Rumbaugh, Bruade, Eric J. Software Engineering: An Object-Oriented Perspective. Wiley, Tuffley, David. How to Write Software Test Documentation Tuffley Computer Services,1996. Blanchette, Jasmin & Summerfield, Mark. C++ GUI Programming with Qt 4 (2nd Edition) Prentice Hall Open Source Software Development Series, Bradski, Gary. & Kaehler, Adrian. Learning OpenCV: Computer Vision with the OpenCV Library O Reilly, Cawood, Stephen & Fiala, Mark. Augmented Reality: A Practical Guide Pragmatic Bookshelf, [8] [9] [10] James, Mike. Pattern Recognition IEEE, 2004 Heijden, Ferdinand van der. Image based measurement systems: object recognition and parameter estimation. New York : John Wiley & Sons, Morris, Tim. Computer vision and image processing Basingstoke : Palgrave Macmillan,

11 2. System Overview In this project there are mainly 4 connected hardware components. Camera and controller are the ones that gathers information and sends it to mobile device. Mobile device works as the brain of the system, since all the data is processed in this part. Heads-up display serves as an output device, since the user can see the output that comes from the mobile device. Camera, which is integrated on heads-up-display device, gathers the visual information through web cam and sends the information to the mobile device frame by frame in order for them to be processed. Controller gathers the choices of the user and it also sends this information to the mobile device. Mobil device, being the brain part of this system, processes the information gathered from aforementioned devices. Detection, tracking and labeling operations are implemented here. The results of the implementations are displayed on the heads-up-display, hence the user will see the output of the program through that device. The overall relationship among devices referring to inputs and outputs can be seen in Figure 1. Figure 1 - Overall Design 10

12 The goal of this application is to combine the real world and the virtual world on a 3d environment in real-time. By detecting, tracking and labeling the predefined objects on the video frames in real time, this application makes the shopping easier for the user. When the user enters the supermarket, not only it gives information which is desired by the user, but also it informs the user about the environment. As a result, the user can be aware of the sections on sale, the locations of the section, or the cheapest, highest quality product without making any effort. 3. Design Considerations In this section, assumptions, dependencies and constraints in the design of this system is explained. Design goals and guidelines are also clarified Design Assumptions, Dependencies and Constraints Assumptions and Dependencies While designing this project, we needed to make some assumptions related to software, hardware and the environment. First of all, our program is intended to be run on Windows OS. If the working platform is wanted to be changed, some simple modifications should be made. Related to hardware, we expect the user to have a mobile device on which he can run the application. Moreover he needs a head-up display to see the environment, the recognized objects in the environment and their labels. Another hardware expectation from the user is a single controller in order for the user to select products from the application interface. We assume that this controller has key buttons and by pressing them the user will control the product selection mode, which can be seen through head up display. As previously said, the application has to be run on a windows OS. Another dependency for the application to be run is that a portable camera integrated on head up display which is connected to the mobile device in order for the user to see the environment and detecting the objects in real time. 11

13 Related to environment, since our aim is to detect, track and label the objects in real time, we assume that our objects to be detected are simple 3D objects. In our environment each section is assumed to have a square marker pattern identifier. We assume that the program starts detecting, tracking and labeling objects as soon as it enters into camera mode. The assumed end users have no specific features, they are assumed to be standard customers in supermarkets Design Constraints Time Constraints The project started on June 2010, as a senior design project. The schedule of the project has been decided at the beginning of the fall semester of academic year. At the end of this semester, a prototype needs to be implemented. The whole work is divided into pieces for each member of the group Pi in order for the project to be implemented efficiently, using less time. After each work process, the work done by each member will be combined and checked together by all of the group members. The project is intended to be finished at the end of May and detailed schedule has been made for this purpose. The detailed schedule can be found in the Gantt charts in the section 8 of this initial design report. Each member will try to follow this schedule, in the case that a member will fall behind the schedule, the rest of the work for that member may be divided for other members Performance Constraints Since the project will contain many 3D models which are to be detected and labelled in real time, performance will be an important issue. Therefore, several number of tests will be applied frequently before going through each step in the project. Moreover, since the project runs in realtime, the recognition process of the objects has to be as fast as possible and the real time camera screen has to be non-frozen. In order to satisfy this constraint, the process running in background of the application needs to be implemented with the fastest algorithms possible. 12

14 Experience of Team Members All of the team members have little knowledge about image processing, pattern recognition and object tracking. Moreover the members are inexperienced on the OpenCV platform. In order to find the best algorithms in each step, we need to implement each of them and by comparing the results, we are trying to find the best algorithm which also should be suitable for real-time applications. Any other tool to be used is a cross platform graphical user interface framework Qt. Since the members do not have much experience on Qt, many graphical user interfaces which can be implemented via Qt are searched and tried in order to choose the best interface for the user Financial Constraints Since the aim of the project is mainly to detect and track objects in real time, we need to have qualified devices which have fast processors and large amount of memory. At first our intend was to implement this project on mobile phones, however working with the mobile phones that we have will reduce the performance and with them it s not possible to work on real time. Thus at this point, since the group does not have a head-up display with embedded camera, the project is prepared with a simple camera and a mobile device namely a laptop and since the group has no controller, simple keyboard will be used for the demonstrations Device Dependent Constraints Since the user is using mobile devices which are volatile, and the environment is a supermarket in which it s not possible to find power supply easily and continuously, the user needs to make sure that the devices are charged enough before shopping Design Goals and Guidelines In this project, we try to follow the KISS (Keep it Simple Stupid) principle. We try to avoid any complexities in our design, but while doing that, we make sure that every necessary part is included 13

15 in it. Our first concern is performance and speed, because detecting and tracking objects in real time needs fast and correct implementation. In order to achieve better performance and for the simplicity, we decided not to use files. Instead, we decided to get the product information from the database in the program memory, which reduces the response time and helps the program to detect and track objects in real time. In addition, as designing the project, we also take the following features into consideration: Reliability Our project will run without any problems or bugs. There will be no consistency problems. Any given set of defined patterns will be identified correctly according to their types, shapes and colors and labelled correctly according to their quality, cost, popularity (if they ire identified to be products) or according to the type of their square marker pattern(if they are identified to be section). The unidentified objects will not be labelled wrongly. Therefore several tests will be applied in order to provide reliability Usability Since this project intends to simplify the user s life, and a standard user is not expected to have information about the program, we are trying to create a user-friendly interface. The operations are clearly set and lists are easy to modify. Giving information about the products in the application will be done by labeling, so labeling will be clear, understandable for a standard user. As a result by looking at the screen on head up display, the user will understand whether the products are on sale, or the section includes the wanted product easily by looking at the labels on them. Moreover the user may use the application in every supermarket since the application is a mobile application running on a mobile device which provides to the user a great usability. 14

16 Portability This project is designed to be platform independent. Although we make our implementations on Windows OS platform, by making simple modifications and handling dependencies, it will be able to work on other platforms. By these simple modifications the application may be run on Linux or MacOS machines. Moreover this simple modifications leads the application to be ported into mobile operating systems such as Symbian, Android and iphone Extensibility This project is designed to be extensible that is, it s open to changes. Whenever a new operation is desired to be added, it can be implemented easily. For example user may want to choose products and categorize them according to other features than cost, quality or popularity. Moreover, whenever another products are wanted to be identified, their information will easily be added to program s memory and the program will start to identify, track and label the newly coming objects. Hence, our project is adaptable to changes and can easily be modified. 4. Data Design In this project, there are predefined interest objects such as products, sections, and salesmen. However, although the project is based on these objects, and is within the framework of an object oriented approach, the inside story differs because of the requirements of the project. Normally it is expected that objects are created first, and necessary actions are applied on them by using their own methods. However, in the scope of this project, objects are not created at first. On the contrary, every action is to recognize, and label them. Since these objects are not actually kept by creating instances because every new frame is a new subject to the application, and the objects are not the actual modified parts (the modification is done on the image), there are two more entities called UnitManager and SystemManager which handle the necessary actions. System Manager is an entity that controls the listing actions in the first mode, and the mode change. In the second mode, 15

17 SystemManager leaves its place to UnitManager which handles the detection and labeling, but still has job at the background such as exiting the application. Following sections give the detailed information about the mentioned data entities Data Description In this project we will keep the information about the data objects in database management systems, namely MySQL. We will have mainly three database tables which are Products table, Section table and Salesman Table. According to the mode (either Product Selection Mode or Camera Mode) those database tables are reached, necessary queries are made, and the achieved results will be used in other parts of the program like labeling, or listing Product Table Product table keeps the information about products that can be seen in Table 1 below. This table includes 11 attributes namely, ProductID, ProductName, ProductCost, ProductCostRate, ProductSellingRate, ProductQualityRate, ProductStatus,ProductSectionName, ProductShape, ProductColor, ProductSign. Each product has a unique ProductID and will be labelled according to its shape-color-sign information achieved from this table. Table 1 - Product Table 16

18 4.1.2 Section Table Section Table keeps the information about sections that can be observed from the Table 2 below. This table includes 5 attributes namely, SectionID SectionName, SectionType, SectionStatus and SectionSign. Each section has a unique SectionID and will be labelled according to its SectionSign information achieved from table. Table 2 - Section Table Salesman Table Salesman Table keeps the information about salesman that can be seen in the Salesman Table below. Since we don t need to separate identified salesman, it will be enough to keep the color information for salesman in just one row. Moreover in our assumptions we stated that salesman have a unique color, which means that no other data object can have the color of the shirts of the salesman.each salesman will not have an ID, moreover they only have 1 attribute namely SalesmanColor. Table 3 - Salesman Table 17

19 4.2. Data Dictionary In this section data design is presented in detail with explanations of all of the entities, their attributes and methods. The referring class diagram can be found on page 22, Figure 2. Additionally the database design can be found in ER Diagram on page 23, Figure Entities Products in detail. As it is stated before, products have 1 1 attributes and below each of them will be explained o o o o o o o o ProductID : This attribute keeps the ID for each product. Its type is integer and each product has a unique ID. ProductName : This attribute keeps the name information of each product. Its type is String and different products may have the same name if they have different cost, quality rate or selling rate. ProductCost : This attribute keeps the cost information of each product. Its type is float. ProductCostRate : This attribute keeps the ranking information of the product according to cost information among other products of same kind in the database. Its type is integer and this information is kept for the sake of performance. ProductSellingRate : This attribute keeps the ranking information of the product according to the selling information (popularity) in that supermarket among other products of same kind in the database. Its type is integer. ProductQualityRate : This attribute keeps the ranking information of the product according to the quality information (whether it s high quality or low quality) in that supermarket among other products of same kind in the database. Its type is integer. ProductStatus : This attribute keeps the information about whether the product is in the Desired Product List or not. Its type is boolean and returns true, if the product is chosen. ProductSectionName : This attribute keeps the information about the section name in which the product is kept. Its type is String. 18

20 o ProductShape : This attribute keeps the shape information of each product.products are 3D geometric objects and their types will be String. o ProductColor : This attribute keeps the color information of each product. Its type is String. o ProductSign : This attribute keeps the information about signs on each product. For the simplicity, the sign on the product will be used to determine its quality rate, selling rate and cost rate Sections As it is stated before, sections have 5 attributes and below each of them will be explained in detail. o SectionID : This attribute keeps the ID for each section. Its type is integer and each section has a unique ID. o SectionName : This attribute keeps the name information of each product. Its type is String and it s also unique for each section, since no two sections have the same name in a supermarket. o SectionType : This attribute keeps the information about whether the section is on sale or not. If the section is on sale, this attribute becomes true and if not, it becomes false. Hence its type is boolean. o SectionStatus : This attribute keeps the information about whether the section includes a product which is on the desired products list. If it includes, this attribute becomes true and if not, it becomes false. Hence its type is boolean. o SectionSign : This attribute keeps the information about the square marker pattern which is a unique identifier for each section. Its type is integer, which means that, the predefined square marker patterns that are kept in the database have ID s which are integers Salesman o SalesmanColor : That s the only attribute of salesman table. It keeps the information about the unique color for the salesman.its type is String. 19

21 UnitManager o ObjectClassifier : That s the only attribute of UnitManager entity. It keeps the information about the type of the object ( product, section, or salesman). According to this information, UnitManager calls different methods SystemManager o Mode : That s the only attribute of SystemManager entity. It keeps the information about the mode of the program whether it s in Product Selection mode or Camera mode Methods Because of the reasons listed in at the beginning of section 4, only UnitManager and SystemManager has methods. The methods that those entities have are explained in the following section Methods Of SystemManager o AddItem() : This method adds the product to the Desired Products List when the user chooses to add it to the list. o ChangeMode() : This method changes the mode attribute of SystemManager entity. With this method, the program leaves the Product Selection mode and goes into Camera Mode. 20

22 o ClearList() : This method deletes all the added items to the Desired Products List. o DeleteItem() :This method deletes the product from the Desired Products List when the user chooses to delete it from the list. o Exit() : This method terminates the application. o GetInfo() : This method writes the information (which is defined in Section 6.1) about the chosen product on the screen when the program is in the Product Selection Mode o Help() : This method opens up a text box that guides the user about how to use the application in the Product Selection Mode. o Start() : This method simply starts the application and loads the Product Selection Mode Methods Of UnitManager o LabelProduct() : This method is responsible for the labeling part of the product. After the products are detected with RecognizeProduct() method, they are labelled according to their quality, cost and popularity rates, which can be understood from the attributes of product entity. This method is also divided into subsections, cause differently identified objects are labelled in a different way. o LabelSection() : This method is responsible for the labeling part of the section. After the section is detected with RecognizeSection() method, it s labelled according to its type (whether the section is on sale or not) and status( whether the section includes product that are in the Desired Products List or not) which can be understood from the attributes of section entity. o LabelSalesman() : This method is responsible for the labeling part of the salesman. They are labelled as salesman after they are recognized with RecognizeSalesman() method. o RecognizeProduct() : This method is responsible for the recognition part of the product. With this method, products are detected according to their shape, color and sign information. This method is divided into subsections, cause different 3D objects are detected with different algorithms. o RecognizeSection() : This method is responsible for the recognition part of the section. With this method section identifiers, which are square marker patterns, are recognized, according to the sign information of the section entity. 21

23 o RecognizeSalesman() : This method is responsible for the recognition of the salesman. It uses only the color information of the salesman entity. Figure 2 - Class Diagram 22

24 Figure 3 - ER Diagram 23

25 5. System Architecture 5.1. Architectural Design The architectural design consists of 4 components, each of which has sub-components and that will be explained in section 5.2 in detail. However, the general architecture design can be seen in the following figure, namely Figure 4. The components are DatabaseHandler, InputHandler, Processor and OutputHandler and they are connected to each other in terms of data flow. Processor, as being the most important part of the design, is the responsible part for image processing operations and user interface. This component gets data that comes from InputHandler and DatabaseHandler, processes those data and sends the results to the OutputHandler. DatabaseHandler component uses Database for information gathering. Figure 4 - Component Diagram 24

26 5.2. Description of Components In this section decomposition of the subsystems in the architectural design is explained in detail. The interaction between the components can be seen as a sequence diagram in the following figure, namely Figure 5. Figure 5 - State Diagram 25

27 DatabaseHandler Processing Narrative for DatabaseHandler The DatabaseHandler component is responsible for accessing database, which is also a subcomponent of DatabaseHandler. When the information about Product, Section or Salesman is needed, the DatabaseHandler component accesses the database, gets the desired information from database and sends it to the Processor component, in order for it to be processed DatabaseHandler Interface Description Since database is designed to be a sub-component of DatabaseHandler, we assume that database is not an input interface to this component. However, DatabaseHandler gets demand from Processor as an input, so Processor is an input interface. The output interface of DatabaseHandler is Processor, because the information achieved via DatabaseHandler is sent to Processor to be processed DatabaseHandler Processing Detail Processor demands for data DatabaseHandler gets this demand as an input DatabaseHandler queries data from Database DatabaseHandler sends the resulting datasets to the Processor component Dynamic behavior of DatabaseHandler DatabaseHandler component has interaction only with Processor. The relation can be 26

28 explained like, when the Processor demands data, the DatabaseHandler gets information and sends the result to Processor, as it can be seen in Figure 6. Figure 6 - DBHandler Component InputHandler Processing Narrative for InputHandler The InputHandler component is responsible for getting data from the environment and the user. With the CameraInputHandler sub-component in it, InputHandler gets the visual data from the environment. It also has a ControllerInputHandler sub-component, which is responsible for getting data from the user. InputHandler is the only component to which the user can directly reach. By using the controller device, the user can make changes on the application. 27

29 InputHandler Interface Description The input interface of this component is the UserInterfaceHandler sub-component of Processor component, because the ControllerInputHandler sub-component of InputHandler waits for the command from UserInterfaceHandler. The output interfaces of this component are Processor s UserInterfaceHandler subcomponent and ImageProcessing sub-component. The gathered visual data via camera is sent to ImageProcessing sub-component, and the data that user gives via controller is sent to UserInterfaceHandler sub-component InputHandler Processing Detail When the program is in Product Selection Mode : UserInterfaceHandler sub-component waits for command from InputHandler. Data is gathered via controller Data is sent to UserInterfaceHandler sub-component of Processor When the program is in Camera mode : Data is gathered via camera The visual data is sent to ImageProcessing sub-component of Processor Dynamic Behavior of InputHandler InputHandler component has interaction only with Processor (with its both UserInterfaceHandler and ImageProcessing sub-components). The relation can be explained like, when the InputHandler gets data via camera or controller, it sends that data to Processor (either to UserInterfaceHandler or ImageProcessing sub-component). The component diagram of InputHandler can be found below in Figure 8. 28

30 Figure 7 - InputHandler Component Processor Processing Narrative for Processor The Processor component is responsible for processing data which comes from InputHandler and DatabaseHandler, and sending the result to the OutputHandler. In both modes, DatabaseHandler brings data from database. When the application is in Product Selection Mode, the data comes from the ControllerHandler sub-component of InputHandler. That data is processed in UserInterfaceHandler sub-component of Processor and is sent to the OutputHandler. When the application is in Camera Mode, the data comes from CameraInputHandler sub-component of InputHandler. That data is processed in ImageProcessing sub-component of Processor and is also sent to the OutputHandler. In the ImageProcessing part, three major operations namely, recognition, tracking and labeling are implemented. 29

31 Processor Interface Description The input interfaces of this component are the DatabaseHandler and InputHandler. Those components send input to both UserInterfaceHandler and ImageProcessing sub-components of the Processor. The output interfaces of this component are InputHandler, DatabaseHandler and mainly OutputHandler. Since Processor demands data from DatabaseHandler and sends wait for command signals to InputHandler, those two components are said to be output interfaces. However, the main output interface of Processor component is OutputHandler, since the processed data coming from both UserInterfaceHandler and ImageProcessing sub-components is sent to OutputHandler Processor Processing Detail When the program is in Product Selection Mode : Processor demands data from DatabaseHandler DatabaseHandler sends data Processor sends wait for command to InputHandler InputHandler sends the data to UserInterfaceHandler, when the controller is used Data is processed The output is sent to the OutputHandler When the program is in Camera mode : Processor demands data from DatabaseHandler DatabaseHandler sends data InputHandler sends the data to ImaeProcessing, when the camera is used Data is processed ( recognition, tracking and labeling operations are made) The output is sent to the OutputHandler 30

32 Dynamic Behavior of Processor Processor component has interaction with DatabaseHandler, InputHandler and OutputHandler. The relation can be explained like, when the data comes to Processor from DatabaseHandler and InputHandler, it is processed and the output is sent to OutputHandler component and it can be seen from the diagram below. Figure 8 - Processor Component OutputHandler Processing Narrative for OutputHandler The OutputHandler component is responsible for showing the output data, which comes from Processor component, on heads up display device. When the application is on Product Selection Mode, OutputHandler gets the data coming from UserInterfaceHandler sub- component of the Processor. When the application is on Camera Mode, OutputHandler gets data coming from ImageProcessing sub-component of the Processor. 31

33 OutputHandler Interface Description The OutputHandler component has no output interface.the input interfaces of this component are Processor s UserInterfaceHandler sub-component and ImageProcessing subcomponent OutputHandler Processing Detail When the program is in Product Selection Mode : o o UserInterfaceHandler sub-component sends the output to OutputHandler OutputHandler handles showing the output on Heads up display device When the program is in Camera mode : o o ImageProcessing sub-component sends the output to OutputHandler OutputHandler handles showing the output on Heads up display device Dynamic Behavior of OutputHandler OutputHandler component has interaction only with Processor (with its both UserInterfaceHandler and ImageProcessing sub-components). The relation can be explained like, OutputHandler gets data from Processor (either from UserInterfaceHandler or ImageProcessing subcomponent), to show the output on heads up display device as it can be seen in Figure 9. Figure 9 - OutputHandler Component 32

34 5.3. Design Rationale As it is stated before, our design is composed of 4 main components, namely DatabaseHandler, InputHandler, Processor and OutputHandler. Since we keep the information about data objects in database, we needed a component for it and we named it DatabaseHandler. Our data objects, whose detailed information is kept in the database tables, are product, section and salesman, so we make them sub-components of database. In this project, gathering information operation is implemented by two different hardware component ( controller and camera) in two different modes ( Product Selection Mode and Camera Mode). However they share same features like they gather information, they send data to Processor component, so we decided to have a component called InputHandler and make ControllerInputHandler and CameraInputHandler its sub-components. In this project, most of the job is implemented by the program itself, without waiting for user s steps. For example object recognition, object tracking and object labeling operations are implemented automatically. So we thought that, we need a component that is responsible for that implementation part and we named it as Processor, since it processes data. We divided that component into two parts namely ImageProcessing and UserInterfaceHandler, since in Product Selection Mode, Processor component is responsible for the implementations which are done on the interface, and in the Camera Mode, Processor component is responsible for processing the images. ImageProcessing sub-section is again divided into three, namely Recognition, Tracking and Labeling, because those three operations are the major image processing implementations of the project. Finally, we decided to have a OutputHandler component since the results coming from both parts of the Processor component need to be shown on Heads-up display device. While dividing the components into sub-components, we tried to divide the system into meaningful parts, however we tried not to divide it unnecessarily. We took into consideration that each sub-component share some features coming from their components, however they have different features than each other. For example, recognition, tracking and labeling sub-components are all parts of image processing, however they are different jobs to be done using different algorithms. As a result we collected them under image processing component, instead of making them components themselves. In the same manner, Controller and Camera are collected under InputHandler component and they are made sub-components. The Aggregation Hierarchy Diagram is following. 33

35 Figure 2 - Aggregation Hierarchy Diagram 34

36 6. User Interface Design 6.1. Overview of User Interface In order to talk about the design of the user interface, one should first define the users. For example, for an application that requires logging in such as the ones used in banks, libraries or even COW, there needs to be different interfaces for different users because every user has to be able to see his authorized actions. To be more tangible, referring to COW, while an admin can ban a student, a student cannot have the authorization to do that, or while an instructor can see the grades of the students, a student cannot see the grades of other students. This kind of features requires different interfaces in order to serve users necessary actions. However, this application, SmARt Shopping, has only one type of user. The user profile for this application is defined to be any person that has the application loaded on his mobile device and wants to do shopping. The application does not have many actors that have distinct privileges while using the application. Thus, there are just one type of user interface which can be observed in two parts. The user interface of SmARt Shopping application has two modes. The first mode is called Product Selection Mode, and this mode welcomes the user. This mode enables the user to list all available products that the application can recognize, choose a product to buy, remove a product from the list that contains the desired products, get help about the usage of the application, and clear the shopping list. In this mode, these actions will be possible by keeping three lists, two of them queried from the database of the application, one for all products and one for desired products that have been chosen by the user. The third list will consist of Clear List, Help, Start Shopping, and Exit actions. Clear List action will clear the list that includes products chosen by the user. Help action will open up a text box that guides the user giving instructions. Start Shopping action will change the mode. Exit action will terminate the application. The user will be able to move inside the lists with up and down keys of the keyboard, and between the lists with left and right keys of the keyboard. The enter and backspace buttons will enable the user to choose a product, and remove a product, in sequence. Enter key will also enable the user to activate the actions such as Clear List, Help, Start Shopping, and Exit. This mode will also enable the user to see some features of the the product that he has chosen before starting shopping such as how many different brands for that product is available on the store, the price range of the product, and the section information of the product. There will be an information section for displaying the features of the current chosen product. The 35

37 current product will be the one that is currently active by the user while moving around. The other mode of the application is not visible until the user is done with selecting products. After the user is done selecting products, the application moves to the second mode which is called Camera Mode. Camera Mode is the part that does the actual job of the application. The actual job is the augmented reality part, which is recognizing the objects and labeling them according to their features. There is not usual graphical user interface in this mode, instead there is a real time video captured by the web cam of the device and the information that is binded on the captured video Screen Images Figure 11 - User Interface Snapshot 1 36

38 Figure 12 - User Interface Snapshot 2 Figure 13 - User Interface Snapshot 3 37

39 Figure 14 - User Interface Snapshot 4 Figure 15 - User Interface Snapshot 5 38

40 Figure 16 - User Interface Snapshot Screen Objects and Actions In the user interface of the application, first of all, a list of available products is available. An example of the available product list can be shown in the first image. The user can see all the available products in this list. Furthermore, the user can move on the items in this list as seen in the second image. When the user stands on one of the items in the available product list and clicks enter, that current item will be added to the shopping list part. For example, in the third image, ketchup is added to the shopping list. When the user continues to move in the list and click enter again, another item can also be added to the shopping list. Forth image is an example of that situation. The user moves in the list and when the coke is the current item, the user clicks enter and now, coke is also added to the shopping list. The user can also move to the shopping list part with right arrow and move in that list. For example, in the fifth image, the user has moved to the shopping list. This mode has one more benefit for the user. When the user is in the shopping list, the information about the current item can also be 39

41 seen in the information area. In the fifth image, when ketchup is the current item, its information is also available in the interface. The user is also able to delect products from the shopping list. In the fifth image, the user was on the ketchup item and by clicking backspace, ketchup can be deleted from the shopping list. The sixth image is a snapshot which is taken after the ketchup is deleted from the shopping list. It can be noticed that, since the ketchup is deleted from the shopping list and coke has became the current item, in the information area, information about the coke can be seen after the deletion. A third part of the list is also available in the user interface. This third part includes four buttons which have some properties. In the seventh image, the user clicks enter when the Clear List button is the current button. By clicking this button, the shopping list can be cleared as seen in the seventh image. When the user clicks enter when the Help button is current, help menu of the application will come to the screen. When the user clicks enter when the Start Shopping button is current, program saves the shopping list and Camera Mode will be opened. When the user clicks enter when Exit button is current, the program will be shut down. 7. Detailed Design 7.1 DBHandler Component Classification DBHandler component is a class with attributes and methods Definiton As it can be understood from the name, DB stands for database, thus, DBHandler component is actually a database handler unit. DBHandler class is used to access database which holds all of the information about data objects of the project. This mentioned data objects are those which are to be recognized, tracked, labeled. 40

42 7.1.3 Responsibilities The primary role of this component is to access database, get the information, and make necessary changes on the database if it is needed. From the accessing point of view, the component queries the needed information from the database. The query demand is made by the processor component, and the resulting data set of the query is sent to again processor component. In brief, this component acts as a bridge between processor component and the database. The database consist of three tables, Product Table, Section Table and Salesman Table. This component uses relational algebra concepts in order to make the demanded query from these tables. Join operators play the major role while making queries in order to get a data set from related database tables, however primitive operations such as projection and selection is also used. For example, processor needs the product name information while dealing with labeling after recognizing the object, and this information is supplied by DBHandler component by making select queries from the relevant table, namely Product Table in this example. Processor may also need the section status of a product in order to inform customer; however, since the processor component is responsible from image processing, only information it can get from the recognized object is the section name. Here DBHandler component steps in and uses join operators between Section Table and Product Table and maintains the data set to the Processor. From the updating point of view, the update demand is again made by the processor. For example while making a shopping list, when the user choses a product, the database should be updated in order to keep the wanted or unwanted information about the product Constraints This component actually plays a major role since its job is maintain data to the Processor, which is the component making the actual job. Since the application is supposed to be real-time, the database will be stored in the mobile device and this will avoid the time loss. There are also contraints referring to database while creating the tables such as not null properties. Since the job is querying, input and output values are limited with one type. Inputs are, strings, turning into queries, and the outputs are again strings, extracted from data sets. 41

43 7.1.5 Composition As it is explained and illustrated in Section 5.2.1, this component has one major subcomponent, the database and the database consists of mainly three tables Product Table, Section Table, and Salesman Table, besides relation tables such as contains, isin, isresponsiblefor tables which can be observed in ER Diagram at the end of Section 4. Database subcomponent is responsible from keeping the information about the interested objects, and the relations between these objects. These tables are the minor, but the but the most essential subcomponents Interactions This component is not reached by the user directly. It will work on background and will be used by Processor. There is no other component that DBHandler component is interacted Resources There are resources that are entities that DBHandler component uses. To begin with, the database itself is a resource for this project. Moreover, in order to keep database, MySQL is used (Detailed informaton about MySQL can be found in Section 8.8.). In order to integrate MySQL with the project, mysql.h header file is used, and it is another resource that this component uses. Since the database is permanent, some non-volatile storage enough to keep the tables is necessary Processing In processing, since this component is responsible from connecting to the database, and making queries. Connecting to the database will be achieved by means of MySQL libraries for C++. A code sample in order to connect to the database is as following: 42

44 Header file is included and necessary variables are defined in order to connect. #include <mysql.h>... MYSQL_RES *result; MYSQL_ROW row; MYSQL *connection, mysql; int state; mysql_init(&mysql); connection = mysql_real_connect(&mysql, host, user, pswd, database,0,0,0); Where host is a host name, the location of the MySQL database server, for example localhost. user is the username pswd is the password. database is the name of the MySQL database reached from host. After connecting to the database there comes the querying part. An explanatory code sample that makes a simple query which brings all of the elements can be found below. The query process is managed in terms of relational algebra rules. state = mysql_query(connection, SELECT * FROM producttable ); if (state!=0) { printf(mysql_error(connection)); return 1; } After the successful execution of the query, we must store the results: result = mysql_store_result(connection); At the end memory should be released in order to prevent too much memory usage: mysql_free_result(result); mysql_close(connection); 43

45 7.1.9 Exports The set of services: Stores data objects Stores relationships between data objects Provides the connection between other components and the database Returns information when demanded 7.2 InputHandler Classification InputHandler is a component with sub-components in it. It can be classified as a class Definiton InputHandler is a component which is designed for controlling the inputs of the system and makes a connection with the Processor component. It is used to handle the inputs which are coming from the controller and the camera. It has two components, namely, ControllerInputHandler and CameraInputHandler. InputHandler component has a relationship only with Processor Responsibilities The main role of the InputHandler component is getting data from the environment and the user. Since SmARt Shopping application has two main parts, responsibilities of the sub-components of the InputHandler are changing accordingly. When the program is in the Product Selection Mode, InputHandler waits input from the controller. When the program is in the Camera Mode, InputHandler is getting the inputs from the captured video frames. 44

46 7.2.4 Constraints The InputHandler component has some constraints related to the quality of the frames captured by the camera. Since InputHandler component gathers the information from the camera and sends it to the Processor to be processed, this frames should be high-quality. In order to have high-quality frames, the system have a high-quality camera. This brings a constraint to the InputHandler component. InputHandler component should have a high-quality camera integrated to the device in order to have high-quality frames Composition As mentioned in the Definition part, InputHandler component has two sub-components which are CameraInputHandler and ControllerInputHandler. This two sub-components are designed for two different modes of the program which are Product Selection Mode and Camera Mode. CameraInputHandler is active when the application is in the Camera Mode and ControllerInputHandler is active when the application is in the Product Selection Mode. When the application is in the Product Selection Mode and ControllerInputHandler is active, this sub-component waits data from the controller. When data is gathered from the contoller, this sub-component sends the data to the UserInterfaceHandler sub-component of the processor. When the application is in the Camera Mode and CameraInputHandler is active, this subcomponent gets data from the camera. This mode is active during the hole Camera Mode. For every frame, input is gathered from the frames and is sent to the ImageProcessing sub-component of the Processor Interactions InputHandler component has an interaction only with the Processor. When InputHandler gets input from the camera by the CameraInputHandler sub-component or gathers input from the controller by the ControllerInputHandler, it sends these inputs to the Processor. Since Processor is the unit which makes processes to these data in order to get results, there should be a component which is feeding the Processor and it is InputHandler. 45

47 7.2.7 Resources InputHandler has two main resources. They are camera which is the resource of CameraInputHandler sub-component and the controller which is the resource of ControllerInputHandler sub-component. The SmARt Shopping application uses two input resources in two different modes and they are the camera and the controller. Since the InputHandler is the component which is dealing with the input, the inputs coming from the camera and the controller is handled by this component. This makes the camera and the controller the resources of the InputHandler Processing The InputHandler component handles the inputs gathered from the controller and the camera. In the Camera Mode, the CameraInputHandler sub-component gathers data from the camera in real-time. While doing this, CameraInputHandler sub-component uses OpenCV s functions and libraries in order to connect the camera and get frames as input. #include "cv.h" #include "highgui.h"... CvCapture *capture = cvcapturefromcam(-1); IplImage * capimg;... while(1) { cvclearmemstorage(storage); cvclearmemstorage(storagepoly); } process_time = (double)cvgettickcount(); capimg=cvqueryframe(capture); // capimg to be processed from now on... 46

48 In the Product Selection Mode, the ControllerInputHandler sub-component is designed to be gather data from the controller. However, it should also be mentioned that in the current application, the inputs of the Product Selection Mode is gathered from the keyboard. Since the UserInterface in this mode is designed with Qt, keyboard functions are also captured with Qt s classes. void MyClass::keyPressEvent (QKeyEvent *key){ switch(key->key()) { case Qt::Key_Escape: break; case Qt::Key_Return: break; case Qt::Key_Up: break; case Qt::Key_Down: break; case Qt::Key_Left: break; case Qt::Key_Right: break; case Qt::Key_Backspace:... break; } QDeclarativeView::keyPressEvent(key); } Exports The set of services: Gathers data from controller Gathers data from camea Sends the data to the Processor component. 47

49 7.3 Processor Component Classification The kind of processor component is module, not class because this component can not be instantiated as objects Definiton The purpose of this component is to handle image processing and user interface operations. In order to do that, processor component gathers information from other components, processes that information and gives outputs Responsibilities The major responsibility of processor component is to process the information. In the product selection mode, this component gets the information from controller subcomponent of input handler component according to the keyboard actions done by the user. Moreover in this mode, information about products, sections and salesmen is gathered from database via database handler component. After combining those information coming from both input handler and database handler, the processor component processes them and serves the output to the user via output handler component. In camera mode, processor component handles the image processing operations, namely object recognition, object tracking and object labeling. The visual information is gathered via camera subcomponent of input handler and recognition, tracking and labeling operations are handled using that information. In order to identify objects and label the identified objects, processor component needs to get the pre-defined information which is kept in the database via database handler. Again after the processing is completed, outputs are sent to output handler component to be displayed on the screen. 48

50 7.3.4 Constraints This project is an augmented reality project, hence the processing data and giving outputs operations need to be handled in real-time. So, timing is the most important constraint for this component and object recognition, tracking and labeling operations should be implemented in realtime. Since main role of this component is image processing, which includes object recognition, lighting is an important constraint. For the video frames to be processed, they should be clear enough. If the light is not sufficient, the objects to be detected may not be distinguishable and it may be difficult for the processor component to identify and track objects. Storage is also a constraint, as a result when the video frames are processed and outputs are sent to output handler component, the pre-processed frames are deleted Composition The processor component is divided into two subcomponents namely image processing and user interface handler. Image processing sub-component has also three subcomponents, each of which are functions, namely object recognition, object tracking and object labeling. Recognition part is responsible for detecting objects on the frames, tracking component tracks the pre-identified objects and labeling component labels the identified objects differently according to their different features. In our design objects to be recognized are products, sections and salesmen. Object recognition subcomponent is responsible for recognizing products according to their shape, color, and sign information, recognizing sections according to their square marker patterns and recognizing salesmen according to their unique color information. After recognition, object tracking component is responsible for tracking all the recognized objects when the frames change. For the user to identify the recognized objects, object labeling component handles labeling operation. When the products are recognized, they are labelled according to their quality, cost and popularity information gathered from database via database handler. Labeling operation is handled for salesmen in the same manner, however this time they are labelled as salesmen, depending on only color information, and all salesmen are to be treated in same way. Moreover, the identified sections which include the desired products, or the sections on sale are labelled differently. 49

51 The second sub-component of processor component is user interface handler. The lists that can be seen on the user interface ( all products list, desired products list) are created with the help of this sub-component. In the product selection mode, while the user is creating shopping list, controller device is used and the information is achieved from input handler component. User interface handler processes that data and handles operations like adding items to the desired products list, deleting items from the product list or clearing the list. It s responsible for bringing the information about the current item on the desired product list via database handler. When the user wants to learn how to use this application, this component is responsible for getting the information text and sending it to the output handler. Moreover, after the shopping list is created and loaded, this sub-component processes the data and sends it to the database handler for the necessary changes to be made on the database Interactions Processor component has no direct user interaction.however, it works as the brain of the system and has interactions with all of the other components of the system. User interface handler sub-component interacts with input handler, output handler and database handler. It works in the product selection mode and gets the information from controller sub-component of input handler. In order to process that data, it refers to the information kept in database via database handler, and after processing this data, it sends the outputs to the output handler to be displayed on the screen. Image processing sub-component also interacts with input handler, output handler and database handler. It works in the camera mode and gets the information from camera subcomponent of input handler. In order for the products, sections and salesmen to be recognized and labelled, their color, sign, shape or square marker pattern information which is kept in the database is needed, so image processing sub-component interacts with database handler. When the objects are labeling operation is completed, the output is sent to the output handler, so the identified and labelled objects are displayed on the screen. Since most of the operations are done in this component and it interacts with all other components of this system, any problems occurred on this component will effect the operations on the other components. 50

52 7.3.7 Resources In image processing sub-component, OpenCV libraries are used to recognize objects. When the objects are recognized, they are labelled with 3d geometric outputs using OpenGL libraries. In user interface part, Qt is the tool that handles the main parts of user interface. Since Processor is the major component in which all the information is gathered and processed, the visual information gathered via camera, the information which is gathered by the demands of the user via controller, and the information coming from database in order to recognize and label the objects according to their features, are all resources that are needed for this component. Depending on the choices of the user, the database may change after loading the list, so processor component may effect the database. Processor component uses memory temporarily, since the frames are kept in the memory while they are processed, and then deleted Processing Image Processing sub-component of this component handles object recognition, tracking and labeling. For object recognition, different methods and algorithms will be implemented for sections and products since their identifiers are different (square marker patterns for sections, geometric objects for products). In order to detect square marker patterns which are identifiers for sections from video frames, the following algorithm is used : First the image is divided into equally spaced regions. Then edges in each region is detected using Gaussian derivative. Sobel filter is used to identify the orientation of the edges. After that line segments in each region is detected using Random Sample Consensus grouping algorithm. This algorithm randomly chooses two points in each region whose orientations are compatible, then the number of points supporting each line is counted. ( the points which lie close to the line and have the compatible orientation with it). After repeating this process several times, the line segments are achieved. Then the segments close to each other and sharing same orientation are merged into lines. After that lines are extracted along the edges until a corner is reached. Since square marker patterns contain squares, only the lines with corners are kept. If the chains of 3 or 4 lines are found, just the chains which form a rectangle counter clock wise are kept. These chains will have black inside and are probably markers. 51

53 With this algorithm, markers are detected, and in order to distinguish them from each other, we put small squares inside the pattern, one for identifying the marker direction and the others representing bits in a binary system. At first, the orientation of the pattern is detected from the square which stands at the outermost line. After the orientation is detected, by means of small black squares in the 4x4 area, location of the black squares distinguishes the square marker pattern from the others. The identification is simply a binary representation. Every square spot represents a bit in a binary system starting from the top-right of 4x4 matrix-like area. Black squares correspond to 1 s, and whites correspond to 0 s. By recognizing squares concerning their locations and perceiving them as numbers, every unique square marker pattern represents a different number. Thus, recognizing and identification is completed. After that, those square marker patterns are labelled, according to the number they represent. An illustration of the marker is following: Figure 17 - Marker Template & Example Representing the Number 10 Since product and salesman objects are 3D objects, 3D object detection should be handled. In order to detect the circular shaped objects, Circle Hough Transform functions which are included in OpenCV libraries are used. To detect circle shaped objects, first the binary image is extracted, then edges are detected. In order to find the centers of the circles, for each pixels on the edges, circles are created, and pixels are converted into ab space from xy space. In the ab space, votes are counted in the accumulator cells. The accumulator cell with the greatest number of votes is the center. Since the edges of the circles are already known, now that the coordinates of center is achieved, the circular shape is detected. After detection they are labelled and their borders are contoured. 52

54 //smooth the image to reduce unneccesary results cvsmooth( gray, gray, CV_GAUSSIAN, 11, 11 ); //get edges cvcanny(gray, edge, (float)edge_thresh, (float)edge_thresh*3, 5); //get circles CvSeq* circles = cvhoughcircles( gray, cstorage, CV_HOUGH_GRADIENT, 1, gray->height/50, 5, 35 ); for( i = 0; circles->total>=1?i<1:i < circles->total; i++ ){ float* p = (float*)cvgetseqelem( circles, i ); cvcircle( src, cvpoint(cvround(p[0]),cvround(p[1])), 3, CV_RGB(0,0,255), -1, 8, 0 ); cvcircle( src, cvpoint(cvround(p[0]),cvround(p[1])), cvround(p[2]), CV_RGB(0,0,255), 4, 8, 0 ); px[i]=cvround(p[0]); py[i]=cvround(p[1]); } The objects which have rectangular shapes are detected using Haar Training. OpenCV library supplies functions to detect rectangular shaped objects. After those objects are recognized, again their borders are contoured. The user interface handler subcomponent of processor compoent makes the suitable implementations after getting the keyboard actions coming from the input handler component. After gathering the information from the input handler, it makes demands to database handler in order to get the necessary information about objects. Then, it sends the outputs of the necessary information to be displayed to the output handler Exports The set of services : Gathers information from input handler and database handler Processes the information Sends the output to the output handler 53

55 7.4 OutputHandler Component Classification OutputHandler is component with one major sub-component in it. This component can be classified as a class Definiton OutputHandler component is the component that handles outputs, namely the user interface and enriched video frames by means of augmented reality. It acts as a bridge between the processor and the graphical output device, namely head-up display in this project Responsibilities This component is actually responsible from delivering the output to the output devices and displaying the outputs. This component takes input from Processor and delivers it to the head-up display which is also the sub-component of this component both in Product Selection Mode and Camera Mode Constraints Since OutputHandler component gives the output to the display device, the aspect ratio of the given output and the monitor should be the same. In other words, if OutputHandler gives outputs in a different size format, the output of the augmented reality and the original world scene would be incompatible. This makes a constraint to the OutputHandler. However, it should also be noted that, when SmARt Shopping is used with the Head-Up Display this constraints will no longer exist. 54

56 7.4.5 Composition OutputHandler component has only one sub-component which is HeadUpDisplayHandler. This sub-component sends the output which is taken from to Processor to the display device, namely, the Head-Up Display. The source of the output can be different. In other words, it can take the output from the UserInterfaceHandler sub-component or the ImageProcessing component. However, although the sources of the outputs are different, HeadUpDisplayHandler does the same thing for every output and sends it to the display device Interactions OutputHandler has an interaction only with the Processor component. After InputHandler component gets the input and Processor makes necessary operations, it sends it to the OutputHandler. When the application is in the Product Selection Mode, OutputHandler gets data from the UserInterfaceHandler sub-component of the Processor. When the application is in the Camera Mode, OutputHandler gets data from the ImageProcessing sub-component of the Processor Resources There is no external entities that this component uses besides Head-Up Display Processing Since there are two types of display, although the job is pretty similar, there are differences. While displaying the user interface, Qt is used (Detailed information about Qt can be found in Section 8.5). However, while displaying video frame outputs, OpenCV functions are used. //necessary actions are done upto this point on the frame image cvshowimage("capture_image",image); 55

57 7.4.9 Exports The set of services: Gathers data from Processor Component Sends it to Head-Up Display and displays the output 8. Libraries and Tools 8.1 OpenCV OpenCV is an open-source library for image and video processing. It s expansion is Open Computer Vision Library. OpenCV is an efficient real time computer vision library which makes itself suitable for most real time applications including Augmented Reality applications that this project implements. It is developed by Intel and now supported by Willow Garage. It focuses on image processing and video processing including real-time video processing which is this project s concern. Since it is written in C, it is portable to different platforms and languages such as C#, Python, Ruby and Java. Since version 2.0, OpenCV also includes the new C++ interface. Much of the new developments of the OpenCV are in C++ interface. OpenCV can run under different Operating Systems (Linux, Mac OS, Windows). 8.2 Eclipse CDT Eclipse CDT is a kind of Eclipse platform that provides fully functional C and C++ Integrated Development Environment. Eclipse CDT includes many features like support for project creation and managed build for various toolchains, standard make build, source navigation, various source knowledge tools, such as type hierarchy, call graph, include browser, macro definition browser, code editor with syntax highlighting, folding and hyperlink navigation, source code refactoring and code generation, visual debugging tools, including memory, registers, and disassembly viewers. 56

58 Eclipse CDT was the appropriate IDE for our project for some main reasons. One of them was, it was compatible with OpenCV and there were examples of linking OpenCV to Eclipse CDT so we have decided to implement our project using Eclipse CDT. Secondly, Eclipse CDT is a cross-platform IDE so that we have Operating System limits. Finally, since we are using QT for creating our User Interface and Eclipse CDT can also be linked with QT, Eclipse CDT was the best IDE for our project. 8.3 MinGW MinGW stands for Minimalist GNU for Windows. MinGW provides a complete Open Source programming tool set which is suitable for the development of native MS-Windows applications, and which do not depend on any 3rd-party C-Runtime DLLs. Primarily intended for use by developers working on the native MS-Windows platform, but also available for cross-hosted use. MinGW includes: A port of the GNU Compiler Collection (GCC), including C, C++, ADA and Fortran compilers; GNU Binutils for Windows (assembler, linker, archive manager) A graphical and a command-line installer for MinGW and MSYS deployment on MS-Windows MinGW is used in this project since the operating system is Windows. Cygwin, an alternate to MinGW is not used because of following reasons mentioned subsequently. Although MinGW forked from version of Cygwin, and both Cygwin and MinGW can be used to port Unix software to Windows, they have different approaches: Cygwin aims to provide a complete POSIX layer on top of Windows, sacrificing performance where necessary for compatibility. Accordingly, this approach requires Win32 programs written with Cygwin to run on top of a copylefted compatibility library that must be distributed with the program, along with the program's source code. MinGW aims to provide native functionality and performance via direct Windows API calls. Unlike Cygwin, MinGW does not require a compatibility layer DLL and thus programs do not need to be distributed with source code. 57

59 8.4 MSYS MSYS is a collection of GNU utilities such as bash, make, gawk and grep to allow building of applications and programs which depend on traditionally UNIX tools to be present. It is intended to supplement MinGW and the deficiencies of the cmd shell. The combination of MinGW and MSYS provides a small, self-contained environment that can be loaded onto removable media without leaving entries in the Registry or files on the computer. 8.5 Qt Qt is a cross-platform application and UI framework. Using Qt, writing web-enabled applications once and deploy them across desktop, mobile and embedded operating systems without rewriting the source code is possible. Intuitive C++ class library, portability across desktop and embedded operating systems, integrated development tools with cross-platform IDE, high runtime performance and small footprint on embedded are the features of QT. Qt uses standard C++ but makes extensive use of a special code generator together with several macros to enrich the language. Qt can also be used in several other programming languages via language bindings. It runs on all major platforms and has extensive internationalization support. Non-GUI features include SQL database access, XML parsing, thread management, network support, and a unified cross-platform API for file handling. Qt is released by Nokia on the following platforms: Linux/X11 Qt for X Window System (GNU/Linux, FreeBSD, HP-UX, Solaris, AIX, etc.) Mac OS X Qt for Apple Mac OS X. Support for applications on top of Cocoa APIs Windows Qt for Microsoft Windows Symbian Qt for the Symbian platform. Qt is to replace Nokia's Avkon as the supported UI SDK for the development of Symbian applications.the Qt for Symbian development group has many quality-controlled articles available. QT Creator and QT Designer is used in this project in order to design our user interface. We also use QT Eclipse Integration in order to integrate Qt to the project. 58

60 8.6 MySQL MySQL is a relational database management system that runs as a server providing multiuser access to a number of databases. MySQL is written in C and C++. Its SQL parser is written in yacc. MySQL works on many different system platforms, including Linux, Mac OS X, and Microsoft Windows. Many programming languages with language-specific APIs include libraries for accessing MySQL databases. These include MySQL Connector/Net for integration with Microsoft's Visual Studio (languages such as C# and VB are most commonly used) and the ODBC driver for Java. In addition, an ODBC interface called MyODBC allows additional programming languages that support the ODBC interface to communicate with a MySQL database, such as ASP or ColdFusion. The HTSQL - URL based query method also ships with a MySQL adapter, allowing direct interaction between a MySQL database and any web client via structured URLs. The MySQL server and official libraries are mostly implemented in ANSI C/ANSI C OpenGL OpenGL (Open Graphics Library) is a standard specification defining a cross-language, crossplatform API for writing applications that produce 2D and 3D computer graphics. The interface consists of over 250 different function calls which can be used to draw complex three-dimensional scenes from simple primitives. OpenGL is widely used in CAD,virtual reality, scientific visualization, information visualization, and flight simulation. It is also used in video games, where it competes withdirect3d on Microsoft Windows platforms. OpenGL serves two main purposes: 1. Hide complexities of interfacing with different 3D accelerators by presenting a single, uniform interface 2. Hide differing capabilities of hardware platforms by requiring support of full OpenGL feature set for all implementations (using software emulation if necessary). OpenGL's basic operation is to accept primitives such as points, lines and polygons, and convert them into pixels. This is done by a graphics pipeline known as the OpenGL state machine. Most OpenGL commands either issue primitives to the graphics pipeline, or configure how the pipeline processes these primitives. In this project OpenGL is used in order to output 3D graphical objects while labeling. 59

61 9. Time Planning 9.1. Term 1 Gannt Chart Figure 18 - Term 1 Gantt Chart 60

62 9.2. Term 2 Gantt Chart Figure 19 - Term 2 Gantt Chart 61