Real Time Tracking System using 3D Vision

Transcription

1 Real Time Tracking System using 3D Vision Arunava Nag, Sanket Deshmukh December 04,2015 Abstract In this report a Skeleton Tracking approach has been described, using the Xbox 360 Kinect camera, as a solution to perform real time tracking of tools and their operation in a manufacturing system. The approach explained in this report describes how the work done by an industrial worker can be mistake proofed, by tracking the worker s skeleton joints in real time. It also proposes a valid approach of how the change in body s positional co-ordinates can be used as a verification for the work performed by the worker. 1 Introduction Nowadays in the manufacturing and automation industries various industrial vision solutions are widely used for purposes like laser cutting, robot automation, quality analysis, and also tracking of various industrial operations. However, the industrial vision solutions are very expensive and most of them are Gray scale and 2D in nature. Hence an economically efficient and easy to use vision solution is required by the industries to replace these expensive yet inferior solutions to perform industrial operations. The manufacturing facility at John Deere- Turf Division wants to mistake proof the operation of ensuring the turf vehicle safety which includes setting the respective industrial standard torques for the bolts on the turf vehicle. There are two kinds of tools used for this purpose presently. One being the sensor-embedded wrench; and the other is the pneumatictriggered tool. These tools though being very expensive, can only provide the torque and related bolt specific data, and lacks any data regarding the location of the tools being used. Hence, a cheap vision solution to perform tracking of the tools to mistake proof the torque fixing of all the bolts in a turf vehicle is required. We proposed to have a Kinect XBox 360 as our vision solution to determine and track the positional coordinates of the worker s skeleton in real time, that would assist the manufacturing execution system to mistake proof the performed tasks. 1

2 2 Kinect Based Vision Solution 2.1 Hardware Details: XBOX 360 Kinect Kinect sensor is a horizontal bar connected to a small base with a motorized pivot and is designed to be positioned lengthwise above or below the video display. The device features an RGB camera, depth sensor and multi-array microphone as seen in figure 1 and the specifications can be found in table 1 [2], [4]. RGB Camera: It stores three channel data in a 1280x960 resolution. This makes capturing a color image possible. Infrared (IR) emitter & IR Depth sensor: The emitter emits infrared light beams and the depth sensor reads the IR beams reflected back to the sensor. The reflected beams are converted into depth information measuring the distance between an object and the sensor. This makes capturing a depth image possible. Multi-Array Microphone: This contains 4 microphones for capturing sound. Because there are four microphones, it is possible to record audio as well as find the location of the sound source and the direction of the audio wave. Accelerometer: A 3-axis accelerometer configured for a 2G range, where G is the acceleration due to gravity. It is possible to use the accelerometer to determine the current orientation of the Kinect. Figure 1: Schematic for Kinect V1 (Source Microsoft Library). 2

3 Table 1: Specifications for Kinect Kinect Features Array Specifications Viewing angle 43 vertical by 57 horizontal field of view Vertical tilt range ± 27 Frame rate (depth and color stream) 30 frames per second (FPS) Audio format 16-kHz, 24-bit mono pulse code modulation (PCM) Audio input characteristics A four-microphone array with 24-bit analog-todigital converter (ADC) and Kinect-resident signal processing including acoustic echo cancellation and noise suppression Accelerometer characteristics A 2G/4G/8G accelerometer configured for the 2G range, with a 1 accuracy upper limit. 2.2 Software Details: Kinect SDK Version 1.8 We used software development kit (SDK) released by Windows to develop the application [1]. Raw Sensor streams: Access to raw data streams from the depth sensor, color camera sensor, and four-element microphone array enables developers to build upon the low-level streams that are generated by the Kinect sensor. Skeletal tracking: The capability to track the skeleton image of one or two people moving within the Kinect field of view make it easy to create gesture-driven applications. 3

4 Advanced Audio capabilities: Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API. Sample code and documentation: The SDK includes more than 100 pages of technical documentation. In addition to built-in help files, the documentation includes detailed walk through for most samples provided with the SDK. Easy installation: The SDK installs quickly, requires no complex configuration, and the complete installer size is less than 100 MB. Developers can get up and running in just a few minutes with a standard standalone Kinect sensor unit (widely available at retail outlets). Design: Designed for non-commercial purposes; a commercial version is expected later. Windows 7 Platform Support: Supports C++, C#, or Visual Basic in Microsoft Visual Studio Skeleton Tracking Analysis In this section we will talk about the what were the challenges and how our approach can be a potential solution to the challenges. We performed tracking of the skeleton in real time using the Kinect sensors. We chose to perform tracking of the worker because of the following challenges: The platform the tracking that needs to be done is small and also the worker blocks the field of view of the camera periodically. Also, the bolts are small enough to be seen and tracked from the given distance of 8-10 feets. To perform position tracking or shift in co-ordinate of the tool from one bolt to other is not possible without using several cameras and enough lighting of the environment is needed, leading to increasing complexity of the operation region. Hence, we chose to track the worker s skeleton joint co-ordinates in real time instead, which can be viewed from given distance at any orientation. At the most, two cameras will be needed to implement this solution, and Kinect is itself a cheap camera as compared to any industrial vision. As we are now only concerned about the worker and his skeleton and hand joint movements it solves the problem of small bolts and the worker blocking the field of view(fov) as well. Kinect being an RGB camera augmented with a depth sensor and IR sensor, it is capable and robust to provide 3D data of the environment within the FOV in normal lighting condition. The algorithm for the developed application flows as shown in figure 2, where the at first the Kinect is initialized and then we fetched the depth data from the sensor. Followed by which the RGB data is fetched as well. Then the skeleton data is extracted where we 4

5 took one frame per second using the Kinect Camera and tracked all the skeleton data that can be seen within the FOV and returned only the closest skeleton, after which we extracted all the joint data and saved them, which are further returned. This returned values are then used to draw the output using the graphics libraries like OPENGL, glew etc. Figure 2: Skeletal Tracking Flow Diagram The application was performed using Microsoft Visual Studio 12 IDE. Below shows the code snippet which performs the skeletal Data extraction written using C++ language. void getskeletaldata() { NUI_SKELETON_FRAME skeletonframe = {0; if (sensor->nuiskeletongetnextframe(0, &skeletonframe) >= 0) { sensor->nuitransformsmooth(&skeletonframe, NULL); // Loop over all sensed skeletons for (int z = 0; z < NUI_SKELETON_COUNT; ++z) { const NUI_SKELETON_DATA& skeleton = skeletonframe.skeletondata[z]; 5

6 // Check the state of the skeleton if (skeleton.etrackingstate == NUI_SKELETON_TRACKED) { // Copy the joint positions into our array for (int i = 0; i < NUI_SKELETON_POSITION_COUNT; ++i) { skeletonposition[i] = skeleton.skeletonpositions[i]; //std::cout<< skeletonposition; if (skeleton.eskeletonpositiontrackingstate[i] == NUI_SKELETON_POSITION_NOT_TRACKED) { skeletonposition[i].w = 0; return; // Only take the data for one skeleton 2.4 Results The tracked skeleton was obtained and viewed using OPENGL API and its libraries along with other graphics processing libraries. The application was designed in C++ to keep it simple and open to any OS platform. It made use of the Kinect SDK and the NuiApi or the Natural user Interface API from Microsoft Kinect. In figure 3 we can see a snapshot from the developed application s output window which shows the 3D point cloud of the FOV and tracks the human s skeleton and joints and marks the hand with the Red Marking line as the application is developed only to track the hands. 6

7 Figure 3: Skeleton Tracking Output Window (Red marks the hand joints) The body orientation was taken in to account and the skeleton could be tracked at any orientation, hence positioning of the camera can be no more an issue. Also, the surrounding environment s depth information has been included which provides the flexibility of tracking of any object or tools in future.the project is an open source hence it is available as a free source (link in section 4)to further development as per needs. Some of the required extension to this application is explained in the section The following code section shows how OpenGL has been used to draw/visualize the Kinect data in the output window. void drawkinectdata() { getkinectdata(); //rotatecamera(); glclear(gl_color_buffer_bit GL_DEPTH_BUFFER_BIT); glenableclientstate(gl_vertex_array); glenableclientstate(gl_color_array); glbindbuffer(gl_array_buffer, vboid); glvertexpointer(3, GL_FLOAT, 0, NULL); 7

8 glbindbuffer(gl_array_buffer, cboid); glcolorpointer(3, GL_FLOAT, 0, NULL); glpointsize(1.f); gldrawarrays(gl_points, 0, width*height); gldisableclientstate(gl_vertex_array); gldisableclientstate(gl_color_array); // Draw some arms const Vector4& lh = skeletonposition[nui_skeleton_position_hand_left]; const Vector4& le = skeletonposition[nui_skeleton_position_elbow_left]; const Vector4& ls = skeletonposition[nui_skeleton_position_shoulder_left]; const Vector4& rh = skeletonposition[nui_skeleton_position_hand_right]; const Vector4& re = skeletonposition[nui_skeleton_position_elbow_right]; const Vector4& rs = skeletonposition[nui_skeleton_position_shoulder_right]; glbegin(gl_lines); glcolor3f(1.f, 0.f, 0.f); if (lh.w > 0 && le.w > 0 && ls.w > 0) { glvertex3f(lh.x, lh.y, lh.z); glvertex3f(le.x, le.y, le.z); glvertex3f(le.x, le.y, le.z); glvertex3f(ls.x, ls.y, ls.z); if (rh.w > 0 && re.w > 0 && rs.w > 0) { glvertex3f(rh.x, rh.y, rh.z); glvertex3f(re.x, re.y, re.z); glvertex3f(re.x, re.y, re.z); glvertex3f(rs.x, rs.y, rs.z); glend(); 8

9 2.4.1 Required Extension to the Application Due to time constraints and lack of hardware availability and not much access to real industry environment the software could not be fully developed to track and mistake proof the bolttorque fixing operation, however they can be implemented easily and has been explained in this section.the present application can perform the skeleton tracking, but some required extension to this application is the tracking of the hand and register the change/shift in the hand joint coordinates and given the threshold of shift, mistake proof can be performed. A certain range of movement from one spot to other spot can be tracked and used for verification purpose, having the practical distance between the bolts in the turf vehicle in to account. This part of the application needs co-ordinate transformation, where the Kinect sensor messages/co-ordinates needs to be transformed in to image co-ordinates (same as done for drawing the skeleton in output window). The algorithm can be developed as shown in flow diagram in figure 4. The positional data can be found using the Kinect API methods as shown below in the code snippet. This snippet is an idea of how it can be done, but it has not been implemented or tested. void performtracking() { getskeletaldata(); NUI_SKELETON_FRAME SkeletonFrame = {0; //left hand float float float hand_left_x=skeletonframe.skeletondata[1].skeletonpositions[nui_skeleton_position_hand_le hand_left_y=skeletonframe.skeletondata[1].skeletonpositions[nui_skeleton_position_hand_le hand_left_z=skeletonframe.skeletondata[1].skeletonpositions[nui_skeleton_position_hand_l float resultant_hand_pos= (sqrt(pow(hand_left_x,2)+pow(hand_left_y,2)+pow(hand_left_z,2))); NuiSkeletonGetNextFrame(0, &SkeletonFrame); float hand_left_new_x=skeletonframe.skeletondata[1].skeletonpositions[nui_skeleton_position_han float 9

10 hand_left_new_y=skeletonframe.skeletondata[1].skeletonpositions[nui_skeleton_position_han float hand_left_new_z=skeletonframe.skeletondata[1].skeletonpositions[nui_skeleton_position_ha float resultant_hand_newpos= sqrt(pow(hand_left_new_x,2)+pow(hand_left_new_y,2)+pow(hand_left_new_z,2)); //finding the absolute value of the co-ordinates float diffvalue=resultant_hand_pos-resultant_hand_newpos; if(diffvalue>6.0) { std::cout<<"mistake Proofed"<< endl; else {//condition 10

11 Figure 4: Hand Tracking Approach Also the refreshing of skeleton frame can be done as shown below in the snippet, where it takes every frame stores the joint information and compare it with the next joint information. void skeletonframecomparison{ NUI_SKELETON_FRAME skeletonframe={0; if (skeleton.etrackingstate == NUI_SKELETON_TRACKED) { // Copy the joint positions into our array for (int i = 0; i < NUI_SKELETON_POSITION_COUNT; ++i) { skeletonposition[i] = skeleton.skeletonpositions[i]; NuiSkeletonGetNextFrame (0, &skeletonframe) for(int j=0;j<nui_skeleton_position_count;++j){ skeletonposition[j]=skeleton.skeletonpositions[j]; difference[]=skeletonposition[i]-skeleton.skeletonposition[j] 11

12 if(difference[j] >=6){ printf( //condition else {//condition //in case the skeleton is not tracked it continues to track if (skeleton.eskeletonpositiontrackingstate[i] == NUI_SKELETON_POSITION_NOT_TRACKED) { skeletonposition[i].w = 0; return; The above snippet has not been tested, and requires implementation. The joint information needs to be taken in to account and then should be fed to the OPENGL class and work with the co-ordinate shifts. 3 Future Work It will be interesting to implement all the required extensions as mentioned in the section and have a working software with those extended methods. Also, it needs to be tested in the real factory environments and figure out if any other challenges that should be taken in to account and perform mistake proofing for the software. Also Industrial Robot Operating System can be used to develop the software for the Kinect, in Linux Platform which provides lot more flexibility and packages to perform development and give the software open avenues to cross platform porting. 4 Conclusion The project was a good learning experience for us. Thanks to our advisor/professor and also John Deere to provide us an opportunity to work on this project. We have performed the skeleton tracking and provided tracking results along with environment information. Also how the application can be fully developed has been explained as well. A video link and the code to to the application has been provided in the Appendix. 12

13 References [1] Microsoft. Kinect for windows sdk v1.8: [2] Microsoft. Kinect for windows sensor components and specifications: microsoft.com/en-us/library/jj aspx. [3] Mircrosoft. Skeleton tracking : [4] WikiPedia. Kinect : [5] Edward Zhang. Kinect skeletal tracking : edwardz/tutorials/kinect/kinect4. html#. Appendix 1. Github Project: 2. Application Video: 13