Development of a Fall Detection System with Microsoft Kinect

Transcription

1 Development of a Fall Detection System with Microsoft Kinect Christopher Kawatsu, Jiaxing Li, and C.J. Chung Department of Mathematics and Computer Science, Lawrence Technological University, West Ten Mile Road, Southfield, MI, , USA {ckawatsu,jli,cchung}@ltu.edu Abstract. Falls are the leading cause of injury and death among older adults in the US. Computer vision systems offer a promising way of detecting falls. The present paper examines a fall detection and reporting system using the Microsoft Kinect sensor. Two algorithms for detecting falls are introduced. The first uses only a single frame to determine if a fall has occurred. The second uses time series data and can distinguish between falls and slowly lying down on the floor. In addition to detecting falls, the system offers several options for reporting. Reports can be sent as s or text messages and can include pictures during and after the fall. A voice recognition system can be used to cancel false reports. 1 Introduction According to the Centers for Disease Control and Prevention [2], falls are the leading cause of injury and death for older adults, with one out of three adults 65 years and older falling each year. As a result, systems that detect falls have become a research topic of interest in recent years. A variety of different approaches have been used to detect falls. Wearable sensors that detect acceleration can be used to detect falls. This approach is examined by Noury et al. [4]. These systems have the drawback that the user must remember to wear the sensors. Floor mounted sensors can also be used. Alwan et al. [1] use floor mounted vibration sensors to detect falls. This system eliminates the need for the user to wear a sensor; however, these systems are often expensive and are complex to install. The present paper focuses on computer vision based systems. Fall detection systems using a variety of vision systems have been developed in recent years. Khan and Habib [3] developed a fall detection system which uses a single camera to detect falls. The system uses background subtraction to isolate the location of a person in the image. Motion gradient data is then used to determine if a fall has occurred. The drawback of this system is the motion gradient of a fall very near the camera will be much higher than that of a fall far from the camera. In order to solve this problem, the 3D rather than 2D location of the person must be known. In the present paper an affordable but reliable way to develop fall detection systems using the Microsoft Kinect sensor is introduced. Rougier et al. [5] have developed a similar system which also uses the Microsoft Kinect sensor. This system also implements both a position based and velocity based algorithm for detecting falls. However, J.-H. Kim et al. (Eds.): Robot Intelligence Technol. & Appl. 2012, AISC 208, pp DOI: / _59 c Springer-Verlag Berlin Heidelberg 2013

2 624 C. Kawatsu, J. Li, and C. Chung the present paper uses the locations of 21 joints during the computation while Rougier et al. use only the centroid location. Additionally, the present paper provides methods for reporting falls and reducing false positive reports. The paper is organized as follows. In Section 2 basic information about the Kinect sensor and the accompanying Standard Development Kit (SDK) is provided. Section 3 discusses the two algorithms used to detect falls. The voice recognition system used to validate falls is described in Section 4. Several methods of reporting falls are introduced in Section 5. 2 Kinect Overview The Kinect contains three types of sensors: a standard camera, an IR camera, and a microphone array. The IR camera detects points projected by a laser and automatically converts them into a depth map. The cameras are calibrated so that the depth map pixels correspond to the pixels in the standard camera images. The Kinect SDK is a free software package which provides a variety of useful tools. The software will automatically detect the 3D location of 21 joints for two people. No markers are required for the software to detect join locations. Information about the algorithm used by this software is provided in [6]. In addition to joint locations, the Kinect SDK also detects the location of the floor plane. 3 Fall Detection Algorithms We developed two algorithms to detect falls using the Kinect SDK. The first algorithm uses only joint position data. This algorithm calculates the distance from the floor to each joint. If the maximum distance is less than some threshold value, a fall is detected. The second algorithm calculates the velocity of each joint in the direction normal to the floor plane. The velocities are averaged over all joints and many frames. If this average velocity is lower (downward velocities are defined as negative) than some threshold value, a fall is detected. 3.1 Position Algorithm The Kinect SDK provides data in frames at a rate of 30 frames per second. Each frame is processed by the fall detection algorithm to determine if the state is fall or not fall. For each joint on a skeleton 4 pieces of data are acquired from the Kinect SDK: the x, y, and z coordinates of the joint and the tracking information for the joint. Joints can be tracked, not tracked, or inferred. The x, y, and z coordinates are Cartesian coordinates in meters with the Kinect sensor located at the origin. In addition to joint information, the equation of the floor plane (in the same coordinate system as the joints) is acquired. The Kinect SDK provides the plane information in the form of the A, B, C,andD parameters from Equation (1). Ax + By +Cz+ D = 0 (1)

3 Development of a Fall Detection System with Microsoft Kinect 625 In Cartesian space, the length of a vector normal to a plane ending at a point (x,y,z) can be calculated using [7] Ax + By +Cz+ D d = A 2 + B 2 +C. (2) 2 Using this relation the normal distance from the floor to each joint is obtained. The fall detection algorithm considers the distances only for joints which are tracked by the Kinect. If every tracked joint has a normal distance less than some threshold the algorithm sets the state to fall, otherwise not fall. Fig. 1. Maximum joint distance from floor plane is above the threshold 3.2 Velocity Algorithm The Kinect provides approximately 30 frames per second of data. From each frame we use the timestamp (in milliseconds) and the 3D Cartesian coordinate location of each joint. We also use the angle of the Kinect sensor which is assumed to not change throughout any calculations. Our algorithm also assumes that the Kinect is placed on a level surface. Instead of using the floor plane equation provided by the Kinect (this is not always detected, particularly on stairs) we calculate our own floor plane. If we assume the Kinect is on a level surface then we can calculate the floor plane equation from the angle of the sensor as follows: where Ax + By +Cz+ D = 0 A = 0, B = cosθ, C = sinθ, D = 3.

4 626 C. Kawatsu, J. Li, and C. Chung Fig. 2. Maximum joint distance from floor plane is below the threshold A, B,andC are simply the vector normal to the floor and D shifts the floor plane 3 meters below the Kinect. The distance from the floor plane can then be calculated using: d = Ax + By +Cz+ D A 2 + B 2 +C 2. For frame i and i + 1 the velocity for a particular joint normal to the floor is then: v i = 1000 d i+1 d i t i+1 t i Where t is the timestamp in milliseconds. The factor of 1000 allows us to work in more convenient units of meters per second instead of meters per millisecond. This velocity is averaged over N frames: v avg = 1000 N 1 d i+1 d i N 1. i=1 t i+1 t i If a joint is not tracked for frame i or i + 1, the velocity is not used and the value of N for the joint decreased. Finally we take v avg from all 20 joints and average again: v jointavg = j=1 v avg, j. If v jointavg is less than -1 meters per second the algorithm detects a fall. This algorithm has problems detecting falls in a few cases. First, if someone jumps in front of the camera it is detected as a fall. This occurs because the speed normal to

5 Development of a Fall Detection System with Microsoft Kinect 627 the floor is very high and the duration of the downward portion of the jump is about the same as a fall. Second, if a person walks out of the Kinect s vision range this is occasionally detected as a fall. The reason for this is because as the person walks off the camera, all of the tracked joints are shifted to the part of the person that is still visible to the Kinect s camera. For example, if only the lower half of a leg is visible to the Kinect, all 20 joints will be tracked within the area of the lower leg. This sometimes causes a very high downward velocity to be detected by the Kinect. Lastly, this algorithm does not perform very well on stairs. In order to detect falls on stairs the threshold velocity has to be lowered. Additionally, cases where someone falls forward when walking up stairs are very difficult to detect. The first problem is not really possible to fix aside from using a different algorithm; however, it is unlikely people using this system will be jumping in front of the camera. For the second case we can eliminate most of the false reports by ignoring cases where all of the joints are very close to the left or right side of the camera s vision. Detecting falls on stairs will most likely require a different algorithm. For example, we could measure the angle of a person s posture while walking on stairs. If this angle deviates greatly from the up direction we detect a fall. 4 Validation Voice recognition is used to reduce false positive reports. After a fall is detected, the event is validated using the Kinect microphone array and a voice recognition system. Once a fall is detected, a new thread is created to ask the user if the require assistance. The thread waits for a response of yes or no. In the case of a yes, a fall is reported. In the case of a no, the report is canceled. A timer is also set. If the timer ends without receiving a yes or no response, a fall is reported. Fig. 3. The GUI when no fall has occured (white border on image)

6 628 C. Kawatsu, J. Li, and C. Chung 5 Fall Reporting Falls are reported through . After the algorithm detects a fall, pictures are taken 15 and 60 frames after the event. These pictures are sent to a user defined address after passing the validation component. These pictures can also be sent to phones using Multimedia Messaging Service (MMS). Most mobile phone providers offer a service which forwards s with attached pictures as MMS messages. This method is free but requires the user to know the form of the address expected by the mobile phone provider. More robust services that only require a phone number are available; however these services are not free and charge for each message sent. Fig. 4. The GUI after a fall has been detected and validated (red border on image) Fig. 5. MMS messages sent by the system

7 6 Experimental Results Development of a Fall Detection System with Microsoft Kinect 629 The system has been tested quite extensively in our lab. All of the cases we observed where the system failed to detect or incorrectly reported a fall have been described in Section 3. One major concern is that a fall simulated in our lab may be significantly different from an actual fall. This could have a large impact on the velocity based algorithm. For example, if actual falls have a shorter duration or lower velocity than those recorded in our lab, the number of frames and threshold velocity would have to be adjusted. Fig. 6. Picture taken during the fall Fig. 7. Picture taken after the fall 7 Concluding Remarks The fall detection system provides an affordable way to detect and report falls. The system has also been tested with people using canes, crutches, and walkers and works reliably. The software is available for download from

8 630 C. Kawatsu, J. Li, and C. Chung References 1. Alwan, M., Rajendran, P., Kell, S., Mack, D., Dalal, S., Wolfe, M., Felder, R.: A smart and passive floor-vibration based fall detector for elderly. In: 2nd Information and Communication Technologies, vol. 1, pp (2006) 2. Centers for Disease Control and Prevention. Falls Among Older Adults: An Overview, Falls/adultfalls.html 3. Khan, M.J., Habib, H.A.: Video Analytic for Fall Detection from Shape Features and Motion Gradients. In: Proceedings of the World Congress on Engineering and Computer Science, WCECS 2009, San Francisco, USA, October 20-22, vol. II (2009) 4. Noury, N., Fleury, A., Rumeau, P., Bourke, A., Laighin, G., Rialle, V., Lundy, J.: Fall detection - principles and methods. In: 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp (2007) 5. Rougier, C., Auvinet, E., Rousseau, J., Mignotte, M., Meunier, J.: Fall detection from depth map video sequences. In: Abdulrazak, B., Giroux, S., Bouchard, B., Pigot, H., Mokhtari, M. (eds.) ICOST LNCS, vol. 6719, pp Springer, Heidelberg (2011) 6. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, pp IEEE Computer Society, Washington, DC (2011) 7. Weisstein, E.W.: Point-Plane Distance. From MathWorld A Wolfram Web Resource,