Optimally Placing Multiple Kinect Sensors for Workplace Monitoring

Transcription

1 Nima Rafibakhsht Industrial Engineering Southern Illinois University Edwardsville, IL H. Felix Lee Industrial Engineering Southern Illinois University Edwardsville, IL Jie Gong Civil Engineering Rutgers University Piscataway, NJ Abstract Real-time range cameras have great potential for automated monitoring of manufacturing work space for safety and productivity improvement. However, multiple range sensors are often needed in order to cover a large manufacturing workspace. Optimal placement of range sensors has great implications on the cost and performance of such automated workspace monitoring systems. In this paper, we use Microsoft XBOX Kinect as an alternative to expensive TOF cameras, and explore the optimal placement of Kinect sensors for monitoring manufacturing workspace. More specifically, the purpose of the research is to develop a heuristic algorithm for optimizing each Kinect s placement in a rectangular manufacturing workspace. Keywords: Microsoft XBOX Kinect, Bin Packing Algorithm, ToF Sensors, Optimal Placement 1. Introduction Manufacturing workspaces have been always places with high potential for danger and accidents. Falls, caught-in-between, and electrical shock are common causes for accidents happening in manufacturing workspaces [1]. The majority of these accidents are rooted in the unawareness of workers performing a certain action such as entering the blind side of equipment or a dangerous area with no caution sign. Extensive research has been conducted on using technologies to prevent dangerous activities or increase workers situation awareness in risky environment. An overarching goal of using technologies in manufacturing environment is to decrease the necessity of manual works. Having various robots or automated machines is a good example of shifting from labor intensive to capital intensive paradigms. However, completely removing human from manufacturing workspace is not feasible in the near future. Manufacturing workspace is dynamic and complex. There is a high potential of being struck by moving objects while working in this environment. Lack of modern technology to control and monitor these kinds of movements is one of the root causes of these types of accidents. There is a need to develop new and better safety features in these places to monitor and predict object movement in manufacturing workspace. Range cameras have been considered to be used for such real time analysis. They have been used for several sensing purposes, but [2, 3] mainly focused on real time site spatial modeling applications, range sensor limitations and their potential for monitoring environments. Besides the range cameras described in these researches, recently Microsoft introduced the Xbox Kinect sensor which was found to be a good alternative for costly vision sensors. Kinect provides improved performance, resolution and accuracy at a fraction of the cost. The majority of the Kinect researches, or even range based sensors, have focused on evaluating the sensor and exploring its application on different environments. In [4] we investigated the depth accuracy and resolution of these Kinect sensors and explored the interference impact between two Kinect sensors in a workspace. Also, in [5] we applied Fuzzy logic to predict sensing performance and determine optimal placement of two Kinect sensors to maximize sensing performance. In this particular research, we mainly focused on using multiple devices in manufacturing places and finding the optimal placement of these devices in order to maximize detection area and also minimize the interference effect between these Kinect sensors. The research findings enhanced our understanding regarding usage of XBOX Kinect sensors for real-time spatial information acquisition, modeling, and monitoring. Such understanding is critical to establishing an intelligent Research Notes in Information Science (RNIS) Volume14,June 2013 doi: /rnis.vol

2 workspace, where the spatial information of machines, materials, tools, equipment, and people can be continuously sensed. 2. Overview of XBOX Kinect and Related Works Recently, a number of hybrid ranging and photo devices have emerged. These devices are categorized under the term 3D range sensors. Examples of such include D-Imager EKL3104, MESA Swiss Ranger, and Canestra. The emitted infrared light hits the objects in the environment and the camera measures the time of flight (TOF) if the lights are bounced back from objects. As a result, these cameras can operate in completely dark environments. The cost of these cameras is typically in the range of several thousand dollars. The Xbox Kinect senor is a motion sensing input device for Microsoft Xbox 360. It is capable of providing streaming depth information and color information at the resolution of 640x480 (much more than comparable devices) and the rate of 30 frames per second. Despite its gaming purposes, Kinect has been found to be an effective solution for spatial sensing at a very affordable price of $150 per unit. Three-dimensional range sensors, due to their real and near real time capabilities, have a good potential in modeling and monitoring different spaces. These sensors have applications in manufacturing workplaces such as collision avoidance [6, 7].The 3D range sensors have usually a limited detection range which would be from 1 meter to about 10 meters depending on the kind of camera. Since the manufacturing sites are normally large and also are often occupied with sizable devices, in some places there would be no clear sight to be detected by one sensor; these may show the importance of using multiple range cameras with high prices (>$5000 per unit) which may not be affordable for everybody. Microsoft changed the situation by releasing Microsoft Xbox Kinect with the price of $150 per unit. The majority of Kinect research published since its release has been focused on introducing, calibrating, and investigating accuracy and precision in various environments [8, 9, 10, 11, 12]. Kinect is also used for detection algorithms [8] and even rehabilitations [13]. Our previous research, [4], focused on investigating Kinect applications on construction and manufacturing sites and [5], proposed a fuzzy logic based model to find optimal placement of two Kinect units with minimum interference effect between. In this paper, we propose models for optimal placement of one device and for multiple devices (more than 2) to maximize the sensing area in a rectangular workplace with known dimensions. 3. Optimal Placement Before we present methods on how to place the Kinect sensors to maximize the monitoring capability, we first report our experimental results on the Kinect sensing range. We conducted experiments at our campus gym to find the indoor sensing range of the Kinect sensor. Figure 1 depicts the sensing range in 2D and 3D we found in our experimental setting. We note that the Kinect sensing range can vary subject to different environmental conditions m 0.26 m 0.49 m 9.25 m Figure 1. Kinect Sensing Range: 2D Area and 3D Volume 730

3 For simplicity of the analyses, we will use the 2D sensing range in this paper. The Kinect sensing area of the 2D drawing is defined by four different lines and curves. In order to explore the optimal location of the device, the equation of these lines must be known. Denoting (A,B) as the Kinect placement location and D the orientation angle, following four equations are extracted from the figure above: Bigger Circle (with the radios of 9.25): Smaller Circle (with the radios of 0.49): Upper Line: Lower Line: We assume that the shape of the detection area is a rectangle, since a rectangle shape is very commonly found in manufacturing and service industry layout settings. 3.1 One Kinect For this section, we have only one Kinect and a workspace with a rectangular shape dimensioned as (L 1, L 2 ). The optimal placement (A, B, D) of the Kinect sensor that gives the maximum detection area depends on the values of L 1 and L 2. However, we can show that the optimal values of A and B remain unchanged as any of the four corner points, regardless of any L 1 and L 2 values. The remaining decision is the orientation angle D, which depends on the rectangle dimensions. Without any loss of generality, we assume L 1 L 2 : Case 1: L 1 and L m: In this case, we can algebraically show that when we place a Kinect sensor at one corner, there is an angle (D) in which 100% of its Kinect detection area can be used. Therefore, the area captured by Kinect would be calculated by: Case 2: : When we place the Kinect sensor in a corner, we can show there are four intersection points between the Kinect detection area perimeter and workplace sides as shown in Figure 2. Below we present a method that determines the Kinect s rotation angle (D) maximizing the detection area. p2 F2 p4 k2 p3 F1 M2 k1 L1 M1 p1 L2 When we denote k 1, k 2, M 1, and M 2 as Figure

4 ( ) { ( ) ( ) ( ) the angles and are calculated as: ( ) ( F 1 and F 2 (two areas out of workspace limits) can be written as below: ) { ( ) ( ) Finally, the area surrounded by sides of Kinect s range of view and workplace walls for a given D, denoted as T(D), can be written as: We can employ single variable search methods to find the optimal rotational angle D that maximizes the detection area T(D). Note that the necessary search range of D is between and = Case 3: L m and L m: Under this case there are only two intersections between the Kinect detection range and the rectangular workspace and we can show the detection area T(D) as [ ] [ ] Case 4: L 1 or L m: * + [ ] ( ) When the workspace has the dimensions of L 1 =8m and L 2 =7m, the optimal location of the Kinect would be at the corner of the site with the rotation angle of to achieve the max coverage equal to 37.56m

5 3.2 Two Kinects More than one Kinect device is needed when the workplace size is larger than one Kinect range of detection or when there are obstacles including manufacturing equipment which can limit Kinect monitoring area. In [4] we conducted experiments to investigate the interference effect between two Kinect sensors. Since the Kinect emits infrared light to get depth data from its environment, the interference of lights emitted from different sensors would led to resolution reduction in captured 3D images. Figure 3 shows how to Kinects were placed during the experiment. Figure 3. Experiment setup with two Kinects In the experiments, two Kinects faced each other with varying angles and distances between them. The minimum angle of 35 was recommended to get an acceptable resolution. The farther distance tends to give the less interference effect. In [5] we used these interference effect data and employed a fuzzy logic in order to determine the optimal placement of two Kinect sensors. We used Matlab fuzzy logic toolbox to determine the angle and the distance between two Kinect sensors while fixing one device coordinates. 3.3 Multiple Kinects This section discusses how to place multiple Kinects to monitor a large workplace while minimizing the number of Kinects needed and the interference effect among Kinects. We transform this difficult problem into a variant of a 2D bin packing problem with some simplification. We replace the circular edge of the 2D Kinect detection area with a straight line. If we do so, the detection area of one Kinect would become a simple isosceles triangle. We consider two types of such triangles: one is making a triangle with the lower bound edge (The triangle ABC in Figure 4) and the other is making a triangle with upper bound edge (The triangle ADE in Figure 4). Lower Bound Edge B D Upper Bound Edge A Figure 4. 2D Kinect Range of View With Lower Bound Edge Method, we are limiting the Kinect range of view to about 8 meters which is 1 meter less that its actual range. This will overestimate the number of Kinects needed. Figure 5 shows a 2D schematic of a W*L workspace, monitored by Kinect using Lower Bound method. Since the bin packing algorithm is typically designed for 2D rectangles, we adapted our problem by sticking two triangles ( abc and cbd ) and making a bigger rectangle ( abcd ). C E 733

6 a b L c d W Figure 5. Workplace Covered Using Lower-Bound-Edge Method The number of Kinects by this method can be calculated by: ( ) ( ) We give an example. For a workplace with dimensions of W=20m and L=50m, a total number of 30 Kinect units are needed to monitor this workplace when we apply this proposed bin packing heuristic algorithm with the Lower-Bound Edge method. Note that the Kinect placement shown in Figure 5 does cause any concern on the interference effect among Kinects that was discussed in Section 3.2. They do not face each other but are offset by 4.34m vertically and sufficiently distanced apart (8.169m horizontally). With Upper Bound Edge Method, we consider the line DE in Figure 4 as upper bound edge for detection area triangle which decreases the accuracy but minimizes the number of Kinects. This method may be useful for places with less required accuracy; places which only want to detect motions not tracking or measuring them. The number of Kinects used in upper bound edge method is calculated as ( ) ( ) When we apply the heuristic algorithm with the Upper-Bound-Edge method, we find the number of Kinects equal to 25 which is 5 units less than the one with the Lowe-Bound-Edge method. Uncovered areas in each method (the white area in the above figures, Figure 5) is divided in two parts; one is a set of small triangles at the bottom of each column and the second is a rectangle on the right hand side of the workplace which has a varying size with different W and L. As we look at the first column of the rectangles of abcd filling from the left upper corner (see Figure 5), we have a partial unfilled rectangle area at the bottom. Let us denote r as its width as shown in Figure 6. Since a real Kinect range is triangular (see Figure 4), the real uncovered area is the white triangle (see Figure 6). We could mathematically show that we could minimize the size of the real uncovered area by shifting down the entire column by r/2. This will create two smaller uncovered triangles, one at the top and the other at the bottom of the column. The rectangle strip leftover on the right side of the workplace can be divided in separate smaller rectangles and each one can be covered using one Kinect optimal placement method which was discussed earlier. r Figure 6. Magnified View of Uncovered Area (White Triangle) 734

7 4. Conclusion The Kinect Sensor, due to its low cost and real time monitoring ability, has a great potential for spatial sensing applications. The performance of the device will be optimized if we locate them in an appropriate place in a manufacturing workspace. In this research we first presented an optimization model to find the optimal placement of a Kinect in a rectangular site. Then we presented a bin-packing based heuristic method to determine the optimal placement of multiple Kinects in order to maximize the detection area while minimizing their interference effects. 5. References [1] "Bureau, Health and Safety Statistics," U.S. Department of Labor, Bureau of Labor Statistics, [Online]. Available: [2] J. Gong and C. H. Caldas, "Data processing for real-time construction site spatial modeling," Auotmation in Construction, vol. 17, pp , [3] C. C. a. C. H. J. Teizer, "Real-time three-dimensional occupancy grid modeling for the detection and tracking of construction resources," ASCE Journal of Construction Engineering and Management, vol. 133, p. 880, [4] J. G. M. S. C. G. H. L. N. Rafibakhsh, "Analysis of XBOX Kinect Sensor Data for Use on Construction Sites," in Construction Research Congress (CRC), Purdue University, Indiana, [5] H. L. J. G. N. Rafibakhsh, "Applying Fuzzy Logic for Optimal Placement of XBOX Kinect Sensors for Industrial Applications," Journal of Advanced Materials Research, no , [6] C. C. a. J. G. S. Chi, "A crash avoidance framework for heavy equipment control systems using 3D imaging sensors," ITCON, vol. 13, pp , [7] C. K. a. K. C. H. Son, "Rapid 3D object detection and modeling using range data from 3D range imaging camera for heavy equipment operation," Automation in Construction, vol. 19, pp , [8] a. M. S. E. Stone, " Evaluation of an Inexpensive Depth Camera for Passive In-Home Fall Risk Assessment," in Proceedings, Pervasive Health Conference, Dublin, Ireland, [9] K. R. Y. S. C. B. A. S. M. M. K. Berger, "Markerless Motion Capture using multiple Color-Depth Sensors," in Vision, Modeling, and Visualization (VMV), Berlin, Germany, [10] T. Dutta, "Evaluation of the Kinect sensor for 3-D kinematic measurement in the workplace," Applied Ergonomics, pp. 1-5, [11] K. Khoshelham, "Accuracy Analysis of Kinect Depth Data," in International Society for Photogrammetry and Remote Sensing (ISPRS), Zurich, Switzerland, [12] T. Stoyanov, A. Louloud, H. Andreasson and A. Lilienthal, "Comparative Evaluation of Range Sensor Accuracy in Indoor Environments," in European Conference on Mobile Robots (ECMR),, Orebro, Sweden, [13] Y. Chang, S. Chen and J. Huang, "A Kinect-Based System for Physical Rehabilitation: A Pilot Study for Young Adults with Motor Disabilities," Research in Developmental Disabilities, vol. 32, pp , [14] S. Q. Xie, G. G. Wang and Y. Liu, "Nesting of two-dimensional irregular parts: an integrated approach," International Journal of Computer Integrated Manufacturing, vol. 20, p ,