An Efficient Magic Mirror Using Kinect

Transcription

1 An Efficient Magic Mirror Using Kinect Md. Moniruzzaman Monir Nahyan Ebn Hashem Md. Nafis Hasan Siddique Afsana Pervin Tanni Jia Uddin Abstract To enhance users shopping experience and to spend less time on queuing for fitting room, this paper presents a virtual mirror model using gesture recognition technique. This allows a person to check how a dress looks like and which color is suitable on a person s body. Moreover, it shows users body measurement when users try on virtual cloths. In the proposed model, we used Microsoft Kinect sensors to track user skeleton movement and depth image. The cloths are simulated using unity engine that will represent an environment for the user like the mirror. In addition, we have developed an algorithm for matching up all the motions between the virtual cloths and the human body. Keywords- Kinect for windows; Gesture recognition; Skeleton Tracking; Real time image Processing; Virtual try-on I. INTRODUCTION In this modern era of revolution everyone is depending more and more on technology. According to this flow of development in technology the common definition of shopping is also changing by time to time. Now the traditional shopping has been replaced by online shopping. Now-a-days online shopping or shopping through web is getting more popular because it is saving huge amount of valuable time of the shoppers and also reducing other hassles. Moreover, online shopping is being accepted widely all over the world. More than 85% of world s population has ordered goods over the internet during the recent years [1]. People are getting more attracted to the online shopping because of its extra features or offers like free home delivery, cash on delivery, and different kinds of discounts. However, it has a significant drawback- this method is not being accepted by all peoples as there is no surety that the delivered goods or cloths will be according to the expectation of the customer. Although customers can find all the description of the cloth like style, size, color fabric and other features through the web page, but they cannot determine whether the cloth is exactly suitable for their own style, color, size and other aspects. Therefore, the delivered clothes might also not fit the customers [2]. Previously, a number of researchers worked on this area to overcome the problems of online shopping. The researchers came up with an idea of virtually try the dresses or clothes so that the user do not have to try it physically [3,4]. Among the works, we have chosen Magic Mirror Using Kinect, by A. B. Habib, A. Asad, W.B. Omar, BRAC University (2015). In this paper authors proposed a model, which has the following limitations: User needs to move to adjust the cloth within his or her body. Dresses are not accurate to the body shape. Use 2D dresses. Use no user interface. To overcome the limitations, we proposed a concept of real time virtual dressing room [3]. As mirrors are indispensable objects in our lives, the capability of simulating a mirror on a computer display or screen, augmented with virtual scenes and objects, opens the door for solving the major drawback in online shopping concept. An interactive Mirror could enable the shoppers to virtually try clothes, dresses using gesture-based interaction [5]. The proposed method has the following features: Use a static position. Varity of dresses. Display size of dresses. Use hand gesture and improved user interface.

2 In this paper, gesture based interaction techniques are used in order to create a virtual mirror for the real-time virtualization of various clothes. Similar to looking into a mirror when trying on new clothes in a shop, we create the same impression but for virtual clothes that the customer can choose individually [6]. For that purpose, we replace the real mirror by a large display that shows the mirrored input of a camera capturing the body skeleton of a person. The use of a hierarchical approach in an image pyramid enables real-time estimation at frame rates of more than 30 frames per second. This paper is organized as follows- Section II provides a detailed overview of proposed model. Section III includes the experimental setup and outcomes of the proposed implementation. Finally, the paper concludes in Section IV. II. PROPOSED MODEL This project mainly focuses on creating a virtual dressing room. This requires real-time tracking of the user skeleton as well as realistic virtual clothing. For the pose tracking Microsoft Kinect sensor is used which gives more complete and accurate tracking of the user pose than the marker based or image feature based tracking which is traditionally used in augmented reality applications. For the clothing we created a set of 3D dress models that can be rendered into the screen. The focus of this project is on realistic interaction and simulation between the user and the virtual clothing. To achieve this, the clothing needs to: Be aligned correctly with the user position and pose. Move and fold realistically. Be realistically rendered into the environment such as ambient lighting. Figure 1 shows a detailed block diagram of system implementation of our proposed model. 4. Position comparison: In this stage our code will compare each array with previous array of frame to identify the rotation of human. 5. Input: At this stage system will save all the raw data for further calculation. 6. Computation: The raw data are compiled and also compute the dress model data. 7. Compare and Coordinate: From previous step s raw values it will compare human model values with dress model values take some point to coordinate them. 8. Display: after doing all this process our system will display the final output. Then it will prepare itself for next input. Figure 2 illustrates the process flow of our proposed model. Figure 2: Process flow of proposed model. Figure 1: System implementation flow. A. System implementation flow 1. User and Kinect: Human body act as input source and by Kinect we take that input. Both Kinect and raw data from user are taken parallel. After that all information passes to the next session as input. 2. Position and Screen setup: After taking raw values from user and Kinect the system measures the position of user and set screen setup. 3. Screen: Screen will process the raw data and change it some frame model and will show into the screen. B. Process flow Process flow is the diagram of how the system will run in the software stage. The description of the diagram will help to understand what is happening inside the system from system start to simulation of virtual cloth. There are some steps that need to be fulfilled to successfully run the complete system. Some of the steps are in loop, as one step needs to be iterating over and over to get the actual result. First, the starting of the program initiates a series of activities. The memories inside the program are allocated for the different part of the system, where initially the memory is free.

3 Second, the system gives the user a view of interactive screen. Here the user will get the options to choose the cloth they want to try on. The Kinect sensors get the measurement and the structure of the user. Third, the system gets the notification or signal that the cloth data for comparing with the user is now available and the other instruction for the system to get the value properly is ready. In between the second and third step there is a step that would be looped for every cloth. Whenever the cloth data is called this steps are to be executed. The data is initialized in the fourth step. The memory gets the allocation for the cloth data and the necessary data from the database is copied in the system. The data that the GPU (Graphics Processing Unit) will use to render the cloth is also copied in the allocated main memory and the GPU memory. Fifth, The Kinect takes action and uses the sensors to get the human body structure and the coordinate of the body. It helps to identify the body structure, positions and coordination of the skeleton point and the depth perception from the Kinect sensor. Sixth, the necessary data taken from the previous steps helps the system to generate a skeleton from the real images. The skeleton gives the system the base of the virtual dress. The virtual dress is then drawn over this skeleton structure. The Unity Engine gives us the option to apply physics and gravitation in the virtual cloth. So the cloth is drawn based on those options we have in an advance dimension than 2D. When the dress is drawn or generated in the screen we will get an augmented reality based approach of the cloth over the human user. Seventh, in between the fifth and sixth steps there is more important and crucial steps that loops around for every bit change in the system. Every time when the human body or the user changes the position or moves a slight the data is passed by the sensors to the system. The data is then copied and the coordinate is given to the main memory and the GPU memory. When the data is in GPU, then the system uses the comparison algorithm to identify the new portion and to generate cloth for the change of the body. The new position is used for new cloth generation. Eighth, to get into this step the user needs to give some specific instructions. If the user wants to change the cloth or the user want to end this session of the system use, and then these steps will be activated. If the user does nothing, then as long as the user is inside the fixed position the cloth will be continued to be generated for every movement change. If the user chooses to change the cloth, then the system will go to the next step. If the system gets the instruction, then the cloth data will be deleted. There will be no data of the user body structure and the cloth data in the memory. In the last step, the memory will be free for the new data of the cloth. The user instruction for changing the cloth with enable the system for going through this step and clear the current cloth data from the memory. If the session is expired by the user, then again memory will be cleared. This memory clear will get the system in a loop which backs the system to the second step. So this is the main loop for the system to start again for change of cloth, refresh the system or for terminate the system session. In between second and ninth step this step is very important. When the memory is free, it means that the allocation of the memory in main memory and the GPU memory will be cleared and the data will be removed for the new cloth and to get the human structure again. III. EXPERIMANTAL SETUP In this section, total overview and summary of the system is presented, which includes basic experiment setup on. The program flow will be explained briefly. In addition, the design of the user interface debugging functionality is also discussed: A. Basic setup 1. Microsoft Visual Studio Microsoft visual studio is an integrated development environment (IDE) from Microsoft. It is an integrated solution which enables the users to develop console and graphical user interface applications along with Windows Forms Applications (WFP), web sites, web applications, web services etc [7]. Visual studio supports almost all kinds of programming languages including built-in languages such as C, C++ [8] (via Visual C++), VB.NET (via Visual Basic.NET), C# (via Visual C#), and F# (as of Visual Studio 2010 [9]). All of these languages are built on top of the.net Runtime (known as Common Language Runtime or CLR) and produce the same intermediate output in Microsoft Intermediate Language (MSIL) [10]. Like any other IDE, it includes a code editor that supports syntax highlighting and code completion using IntelliSense for variables, functions, methods, loops and LINQ queries [11]. IntelliSense is supported for the included languages, as well as for XML and for Cascading Style Sheets and JavaScript when developing web sites and web applications [12][13]. Autocomplete suggestions appear in a modeless list box over the code editor window, in proximity of the editing cursor. In Visual Studio 2008 onwards, it can be made temporarily semitransparent to see the code obstructed by it [11]. The code editor is used for all supported languages. 2. Microsoft Kinect Kinect is a motion sensing input device developed by Microsoft in early 2010 [6]. Kinect is mainly used in Xbox 360 console and for Windows PCS [6]. For Windows, a Kinect sensor consists of an RGB camera which can store up to three channel data of resolution It has an IR (infrared) emitter which emits lights beams and an IR depth sensor that reads the reflected beams and process the information to measure the distance between the object and the sensor. It also has multi-array microphone that can capture sound and detect the location of the source and direction of the audio wave. It has a practical ranging limit of 40cm 3.5m for Windows and the frame rate is 30 FPS (Frames per Second) [14]. Figure 3 depicted a picture of a Kinect.

4 Figure 5 shows the application flow of the proposed model. Figure 3: Kinect components ( /jj aspx.) Kinect SDK is developed to enable developers to develop applications in C++, C# or Visual Basic by using Microsoft Visual Studio [4]. It is capable of capturing front body 2D motion, gesture, facial and voice recognition [6], skeletal tracking and advance audio capabilities [15]. To setup virtual mirror we need Kinect sensor to record skeleton and depth data and capture the RGB video stream. In addition, the software has the capability to recognize and track human body. The software runtime converts depth data into about 20 skeleton joint points of human body to track up to two persons in front of the camera [16]. Figure 4 shows the joint points of a human body detected by Kinect [18]. Figure 5: Application Flow of Proposed Model 5. Ordinary Differential Equation(ODE) For calculating the velocity and the force of the particles in each frame, differentiation is required. The followings are the 3 equations that were chosen for the calculation of differentiations. Euler Method [19]: y n = y n + hf (t n, y n ) Heun s Method [20]: y n+1 = y n + hf (t n, y n ), y n+1 = y n + h 2 (f (t n, y n ) + f (t n+1, y n+1 )) Figure 4: (a) Skeleton joints found by Microsoft Kinect ( (b) joint structure on avatar [18] 3. Computation To virtually set the cloths on the user and display it in real time first we need to detect the skeleton of the user and record both skeleton and depth data for processing. After that we need to define key joint points using skeletal tracking algorithm and compute the right simulation of virtual cloth to display the right 2D cloth. Lastly, we combine real time video and clothing simulation on skeletal points and displayed it. 4. Display/Screen To give the user a mirrored image impression there is a screen or display in front of the user. Data processing has to be accurate and mirroring the real time video essential in order to give the user an impression of mirror. Runge-Kutta Method [21]: y n+1 = 1 6 (k 1 + 2k 2 + 2k 3 + k 4 ) k 1 = hf (t n + y n ) k 2 = hf (t n h, y n k 1) k 3 = hf (t n h, y n k 2) k 4 = hf (t n + h, y n + k 3 ) After testing on the three differentiation method, only Runge-Kutta method is suitable for the program. The other two methods have relatively big error range of the result and will make the cloth become unstable. B. Full application flow The user will stand in front of the virtual mirror. Then the user will perform a 180-degree calibration pose so that Kinect could record the 3D depth image, skeleton and joint points for calculating the structure of human body. After executing calibration and recording measurement data, the system will continue. Now by using Kinect's skeletal tracking algorithm, it is now possible to access the skeleton parameters and joint point. Now an algorithm needs to keep the track of the change in key joint points to track movement of the user. If any problem

5 occurs during the calibration process, the system will restart. The virtual mirror (screen) will display user s real time mirrored videos. The user will then choose the cloth that the user wants to see virtually in the mirror. The dynamic cloth will appear over the user s garment. The user will have to maintain a minimum distance to track by the Kinect to get the skeleton points and depth data. After the simulation the user will have to move out of the mirror area or sensor s view point to reset the whole system. D. Debugging: Figure 7: Minimum distance for tracking [1]. In the process of debugging the system, we need to get the live value from out testing. The debugging process can be differentiate into two steps: 1. Checking value. 2. test. Figure 6: Kinect setup and human ditection. C. User interface As this is a virtual mirror and it can be implemented anywhere from commercial cloth store to personal dressing mirror, there is no need of external display for the application interface. The virtual mirror is used to show the interface and it is controlled by gesture recognition feature of Kinect. Particularly swipe gesture will be implemented and will be distinguished by right and left swipe method. This swipe method can be tracked and implemented by using the Kinect sensor. By using this feature user will be able to choose their desired cloths to display. Figure 7 indicates that the users would have to be at a minimum distance from the Kinect to be able to successfully track by Kinect [1]. Checking Values In the debugging process, we first need to compile and create the environment of the test to check if we are getting the proper skeleton value and the RGB images. We need these values to do the human tracking that will enable us to create dynamically generated clothes for our system. The values that we would get by calculating the distance of every person s body structure from the skeleton and real images will help us to determine our cloth structure and the stretch point of our cloths. Using these values, we can determine the efficiency of our system. Testing The testing of the code and the system is another important part. We need to test the system using live input to detect the movement and then apply the virtual cloth simulation. From the testing it and we will be able to identify the proper simulation and error in the system. As we have implemented and designed the proper system with minimum error hence the testing will give us the sufficient data to analyze and calculate our virtual mirror effectiveness. In the experimental part, we have generated a skeleton and real image based compound display to identify the movement. There is some of our experimental images that show that we have successfully implemented the code and we can track the movement of the human body. Currently this is a 2D structure based model where we can implement 2D cloths and we will develop it in 3D structure model so that we can implement our main goal of 3D cloth simulation and the virtual mirror. In addition, detection of skeletal joint points are shown in Fegure7.

6 on of garment. The Microsoft Kinect offers the optimal technology for a successful implementation. Compared to other technologies like augmented reality markers or real-time motion capturing techniques no expensive configurations and timeconsuming build-ups are required. From this point of view, it is an optimal addition for a cloth store. A simple setup of the system can also be assembled at home since the minimum requirements are a computer with a screen and a Kinect. (a) (b) Figure 8: Body Simulation and Tracking with 2D cloths. IV. CONCLUSION In this paper, we introduce a virtual dressing room application where avatar and cloth generation, real time tracking technologies up to an overview of comparable virtual try-ons. Subsequently a closer look on the technologies and frameworks that were used for the implementation of the virtual dressing room was taken. After this the different aspects of the design process up to the construction of the garment models was highlighted. This is followed by the implementation, describing the cloth colliders and the behavior of the garment, for instance. In the last section the tests were executed, also discussing the output, the appearance and the interaction with the virtual dressing room. Overall, the presented virtual dressing room seems to be a good solution for a quick, easy and accurate try- REFERENCES [1] U. Cheema, M. Rizwan, R. Jalal, F. Durrani, N. Sohail, The Trend pf online shopping in 21st century: Impact of enjoyment in tam model Asian Journal of Empirical Research, vol. 3, no. 2, pp [2] L. Zhao, J. Zhou, Analysis on the Advantages and Disadvantages of clothing Networking Marketing International Journal of Business and social science, vol. 6, no. 4(1), pp , (2015). [3] A.B. Habib, A. Asad, W.B. Omar, Magic Mirror Using Kinect, BRAC University (2015). [4] S. Giovanmi, Y.C. Choi, J. Huang, E.T. Khoo, K. Yin, Virtual try-on using Kinect and HD Camera, MIG-2012, vol. 7660, pp , (2012). [5] P. Presle, A Virtual Dressing Room based on Depth Data, Vienna Uninversity of Technology, Klosterneuburg, pp , (2012). [6] D. Ravì. Kinect: the next generation of motion control. Retrieved from [7] H. P. Halvorsen, (2014, March 12). Introduction to Visual Studio and C#. Retrieved from n%20to%20visual%20studio/introduction%20to%20visual%20studio% 20and%20CSharp.pdf. [8] Brenner, Pat (19 July 2013). C99 library support in Visual Studio 2013, Visual C++ Team Blog. Microsoft. [9] D. Chai, and K. N. Ngan, Face Segmentation using Skin-Color Map in Videophone Applications, IEEE Transactions on Circuits and Systems for Video Technology, vol. 9, no. 4, pp , (1999). [10] A. Sur, Visual Studio 2012 and.net 4.5 Expert Development Cookbook, vol.1, Chapter No. 1 Introduction to Visual Studio IDE Features, ( 2013). [11] Guthrie, Scott. Nice VS 2008 Code Editing Improvements, July 28, [12] Guthrie, Scott. VS 2008 JavaScript IntelliSense, June 22, [13] Guthrie, Scott. VS 2008 Web Designer and CSS Support, July 25, [14] Kinect for Windows Sensor Components and Specifications. Retrieved from [15] H. Fairhead, All About Kinect,. Retrieved from [16] G. Yolcu, S. Kazan, and C. Oz, Real Time Virtual Mirror Using Kinect, Baikan journal of Electrical & Computer Engineering, vol. 2, no. 2, pp , (2014). [17] A. Kar, Skeletal Tracking using Microsoft Kinect the Microsoft Kinect sensor, Methodology (2010). [18] M. Kotan, and C. Oz, Virtual Mirror with Virtual Human using Kinect Sensor, 2nd International Symposium on Innovative Technologies in Engineering and Science, Karabuk University, Turkey, pp , (2014). [19] Paul s Online Math Note. Retrived from [20] Department of mathmatics, University of Kansas State. Retrived from [21] Erik Neumann, Retrived from