Ready-to-Use Activity Recognition for Smartphones

Transcription

1 Ready-to-Use Activity Recognition for Smartphones Pekka Siirtola, Juha Röning Computer Science and Engineering Department P.O. BOX 4500, FI-90014, University of Oulu, Oulu, Finland Abstract In this study, every day activities are recognized from data collected using smartphones accelerometer sensors. Offline experiments are made to show that the presented method is user- and body position-independent. In addition, it is shown that the features used in the classification are not dependent on the calibration of the phone. The recognition models trained using the offline data are also tested online. A mobile application running these models is built for two operating systems: Symbianˆ3 and Android. Real-time experiments using these applications are made to show that the presented method can be implemented to any operating system and hardware variations do not affect recognition results. Quite high recognition accuracies are obtained, in the offline study, the average recognition rate is almost 99% and, also, in the online study, the average recognition accuracy is over 90%. Index Terms Accelerometer, mobile phones, activity recognition, machine learning I. PROBLEM STATEMENT AND RELATED WORK Nowadays, almost every one has a smartphone and people have them almost all the time with them. Therefore, smartphones are excellent devices to monitor every day activities of people. In this study, every day activities are recognized from acceleration data collected using a mobile phone s sensors. In our previous study (see [1]), we showed that activities can be recognized accurately, in real-time and user-independently using mobile phones. Limitation of that study was that it was not body position-independent, the phone needed to be at trousers front pocket in order to classify movements correctly and reliable. Obviously, the recognition would be much more practical if it would not be dependent on the phone position, which is studied is this article. However, when positionindependent recognition was started to study for this article, challenges were faced. The offline recognition rates were quite high and promising, but when a mobile application was built d based on the trained models, the classification produced weird results and the accuracy was very low. In addition, when the mobile application was tested on multiple devices, different results were obtained with different devices. It was noted that this was caused by differencies in hardware and calibration. Although the phones were of the same model, still there was some differences in accelerometers, and they did not give the same output. This was not an issue in our previous study because the classification process was easier. However, position-independent recognition requires more accurate features and models and therefore hardware variations needed to be considered. Other research have noticed the hardware variation problem as well, in [2] accelerometers were calibrated in order to compensate hardware variations. The study offers two ways to calibrate sensors, a user driven and automatic method. In the user driven calibration method, the user is told to hold the phone on a certain position, and based on this information, variations in sensor values caused by different accelerometer alignments can be calculated. The automatic calibration method tries to find samples where the phone is stationary, and calibration is done based on these samples. According to the study, it takes approximately one or two days to collect samples which can be used to calibrate sensors. Moreover, in the article, five different activities are recognized from the accelerometer data collected using mobile phones. The recognition method presented in the study is body position independent, so the method recognizes activities regardless of the mobile phone s position. Even then, the recognition rates are quite high, 95% of the activities can be recognized correctly in the offline experiments. There are also several other studies ([3], [4], [5], [6], [7]) where the data for activity recognition is collected using a mobile phone but the activity recognition algorithms are not implemented on the phone, and the classification is not done in real-time on a mobile phone as in our study. An activity recognition system running on a smartphone is presented in [8]. The presented system can be trained on the device and does the classification in real-time on the device. The recognition is based on features calculated using geometric template matching. Support vector machine is used as a classifier. Unfortunately, the study does not include recognition rates: thus, the evaluation of the system is difficult. However, the smartphone application is available from Google Play. The system described in [9] can also be found from Google Play. It seems to recognize activities with accurately, but all the features used in classification are not phone orientation independent. This paper present an activity recognition method that takes into consideration hardware variations. However, it does not require calibration, user inputs are not needed and a mobile application based on the method is ready-to-use straight after installation. Instead, we focus on considering hardware variations already in the feature extraction phase. In addition, it is shown using offline tests that these features can be used to rec-

2 ognize human activities body position- and user-independently. What is more, the real-time experiments show that the presented method can be implemented to every smartphone where the maximum sampling rate of the accelerometers is at least 40Hz. The paper is organized as follows: Section II presents the used data sets. Section III introduces the methods and Section IV evaluates the accuracy of the proposed method in offline and online scenarios. Finally, the conclusions are in Section V. II. DATA COLLECTION The data were collected using a Nokia N8 smartphone [10] running Symbianˆ3 operating system. N8 includes a wide range of sensors: tri-axis accelerometer and magnetometer, two cameras (12 MP and 0.3 MP), GPS, proximity sensor, compass, microphones and ambient light sensor. Still, only the triaxis accelerometer was used in this study to detect activities. Different phones have different hardware, and therefore, the maximum sampling rate at smartphones can differ. Thus, in order to achieve mobile phone independent results, the highest possible sampling rate was not used. Although accelerometers were running at full speed during the collection of the training data, all the samples were not saved nevertheless, and therefore, not used in activity recognition process. Instead, the latest value from the accelerometer was called at every 25 milliseconds and stored to a file. Therefore, the used sampling frequency was 40Hz, which is much less than the maximum sampling frequency of most phones. The used method enables the same sampling frequency to any smartphone, where the maximum frecuency of the accelerometer is at least 40Hz, making recognition less phone model dependent. The classification models used in this study were trained based on the activity data collected from seven healthy subjects. The subjects were carrying five phones at the same time. They were located at trousers front pocket, jacket s pocket, at backpack, at brachium and one at the ear. The participants performed five different activities: walking, running, cycling, driving a car, and sitting/standing. The reason for selecting these activities for the study is that normal everyday life consists mainly of these five activities. Howver, there is no data from each activity from each body position, see I. For instance, subjects were not allowed to cycle while holding a phone at the ear because of safety issues. Moreover, data were collected when a phone was laying on the table. Therefore, six activities were recognized. The total amount of the data collected was about fifteen hours. When the selected activities are studied in more detail, it can be noted that walking and running are different from the other three because everyone has a personal walking and running style. Other activities are not personal, for instance, while cycling, the movement trajectory is predefined. Therefore, the models to recognize walking and running are most challenging to train. The training data were collected by seven subjects whose age varied from 24 to 34 years (average 29 years) and height TABLE I ACCELEROMETER DATA WERE COLLECTED FROM FIVE DIFFERENT BODY LOCATIONS AND FROM SIX DIFFERENT ACTIVITIES. Body position trouser s pocket jacket s pocket backpack ear brachium table Activities walking, running, cycling, sitting/standing, driving a car walking, running, cycling, sitting/standing walking, running, cycling, sitting/standing walking walking, running, cycling, sitting/standing on the table from 1.65 to 1.85 meter (average 1.78 meter). They performed most of the activities outside the laboratory. Subjects walked inside and outside, mainly on flat surface, but also in a staircase. Streets where subjects walked, run, drove a car, and cycled were normal Finnish tarmac roads, and the route and speed were determined by subjects themselves. In addition, the phone orientation was determined by subjects. The roads used for collecting driving a car data included motorways, as well as roads at the city center. Partly the same roads were employed in offline and real-time tests. Sitting consists mostly of office working. In addition, the real-time classification on the device was tested to show that the presented method is not dependent on hardware and operating system. For this experiment, the activity recognition application was implemented to Symbianˆ3 and Android -operating systems. Symbianˆ3 tests were made using Nokia N8 smartphone and Android tests using Samsung Galaxy Mini smartphone running Android operating system. Galaxy Mini is a low budget smartphone having triaxis accelerometer, proximity sensor, compass and 3.15MP camera. It uses 600MHz ARMv6 processor. A. Aim of the study III. METHODS The activity recognition system presented in this study is ready-to-use. This means that the user does not need to set any parameters or do any other actions before he/she starts to use it. Therefore, the system presented in this study is user-independent. In addition, the study uses calibration and orientation independent features, and therefore, the system can be used regardless the hardware differences. Moreover, unlike in our previous study, the user does not need to carry a phone on trousers pocket. Because training data was collected from five different body locations, the presented method supports five different body locations, and therefore, it can be said that it is body-position independent. What is more, using the presented method, an activity recognition system can be implemented to any smartphone, where the frequency of the accelerometer is atleast 40Hz, making the system mobile phone independent. B. Activity recognition Activity recognition process consists of four stages: preprocessing, feature extraction, model training, and classification.

3 In this study, the aim is to do the whole classification process on the device in real-time. Therefore, the recognition process needs to be light so that it does not require a lot of processing power and does not use a lot of battery. 1) Sliding window: The online activity recognition was done using a sliding window technique. The signals from the sensors were divided into equal-sized smaller sequences, also called windows. From these windows, features were extracted and finally the classification of the sequences was done using models trained based on these features. In this study, the windows were of the length of 300 observations, which is 7.5 seconds, because the sampling frequency was 40Hz. In offline recognition, the slide between two sequential windows was 75 observations, while in online recognition, the slide was set to 150 observations to load the processor less. Moreover, to reduce the number of misclassified windows, the final classification was done based on the majority voting of the classification results of three adjacent windows. Therefore, when an activity changes, a new activity can be detected when two adjacent windows are classified as a new activity. For instance, if the slide is 150 observations, a new activity can be detected after 450 observations, which is around eleven seconds if the sampling rate is 40Hz. 2) Feature extraction: In order to achieve reliable phone orientation independent results, the effect of gravitation needs to be elimanated from the sensor values. There are two ways to do that: (1) by recognizing the orientation of the smartphone, or (2) by eliminating the orientation information. On the other hand, the orientation is impossible to recognize using only accelerometers. Therefore, the effect of orientation had to be eliminated. In the preprocessing stage, all three acceleration channels were combined as one using square summing. This way orientation independent magnitude acceleration was obtained. Moreover, the orientation of the phone has some limitations, for example the screen or the back of the smartphone is always against the user s leg when the phone is in the trousers pocket. Therefore, it was tested if features extracted from a signal where two out of three acceleration channels were square summed would improve the classification accuracy. A 3D accelerometer is a sensor consisting of three accelerometers that are approximately perpendicular to each other. However, in reality, in each phone these sensors are aligned a bit differently. Therefore, the output data of the sensors is dependent on the used phone making also the classification result dependent on the phone. Figure 1 shows a magnitude signal from three different N8 smartphones when the phone is still on the table. Average values vary from 9.1 to 10.0 m/s 2 and this variation causes misclassifications. In order to achieve the highest possible detection rates, the methods presented in the study take into considedation these hardware variations. However, unlike previous studies ([2]), it does not require calibration. Instead, we focus on considering hardware variations in the feature extraction phase. Therefore, user inputs are not needed and the mobile application based on the method is ready-to-use straight after installation. Because of the different calibration, accelerometer values from different phones are at a different level. Therefore, the shapes of the signals are the same but absolute values differ by some constant. This difference is normally fixed by using automated or user-driven calibration. However, because the difference in the signal level is the only major difference caused by different calibration, in this study, differences are eliminated simply by subtracting the mean of the window s values from each value of the window. This way, calibration differences can be eliminated already in the feature extraction phase and calibration is not needed. 19 features were extracted from magnitude signal and from the signals combining two out of three acceleration channels after calibration differences were eliminated, so together 76 features were extracted. These features were standard deviation, minimum, maximum. In addition, instead of extracting percentiles, the remainder between percentiles (10, 25, 75, and 90) and median were calculated. Moreover, the sum of values above or below percentile (10, 25, 75, and 90), square sum of values above of below percentile (10, 25, 75, and 90), before summing and number of crossings above of below percentile (10, 25, 75, and 90) were extracted and used as features. Note that the mean was not one of the features because it is zero for every window caused by eliminating calibration differences. 3) Classification: The classification result was obtained using the decision tree presented in Figure 2, which classifies activities using a three stage procedure. In the first classification stage, a model is trained to decide if the studied subject is currently active (walking, running or cycling) or inactive (driving a car, sitting, standing or a phone on the table). After this, one or two more classifications are needed to perform in order to obtain the exact activity. Therefore, five binary classifiers are needed to train. These models were implemented to a smartphone and also used in online tests. Based on the results of our previous study, QDA (quadratic discriminant analysis) was decided to use as a classifier. QDA combines features to find quadric surfaces that separate the classes best. The resulting combination may be employed as a quadric classifier [11]. In order to achieve the highest possible recognition rates, the most descriptive features for each model were selected using a sequential forward selection (SFS) method [12]. Moreover, to obtain reliable user-independent results, the training was performed using the leave-one-out method, so that each person s data in turn was used for testing and the rest of the data were employed for model training. IV. EXPERIMENTS The experiments section comprises two parts: offline recognition and online recognition. The purpose of the offline recognition is to show that the presented method is userindependent and, in addition to showing that it is body position-independent. The models used to offline recognition are then implemented to a mobile phone and online experiments are made to show that the presented method is reliable regardless of the operating system and the hardware variations of the mobile phones. Moreover, experiments are made to

4 Fig. 1. Different phones have different calibration. Fig. 2. The decition tree obtained to recognize the type of activity. TABLE III OFFLINE EXPERIMENTS SHOW THAT ACTIVITIES ARE RECOGNIZED WITH HIGH ACCURACY FROM EVERY BODY LOCATION. Body position Recognition rate trouser s pocket 99.5 % jacket s pocket 97.3 % backpack 97.8 % ear 96.7 % brachium 98.9% table 98.7 % show that the method presented is light enough to be used for real-time recognition. The models were trained using the QDA classifier and they were based on the data set presented in Section II. Online experimets are performed by a person carrying six different smartphones. A. Offline recognition The offline results are based on the data presented in Section II. Model training was performed using the leave-one-out method, so that each person s data in turn was used for testing and the data of six persons were employed for model training. 1) Results: User- and body position -independent results are presented in Table II. To compare how activities are recognized from different body positions, the recognition accuracy from each body position was obtained and the results are presented in Table III. 2) Discussion: Offline experimets show that using the presented method activities are recognized with a high accuracy user-independently, see Table II. In addition, all the activities are detected almost perfectly from each body-position. Moreover, the average recognition rate (98.9%) is even higher than in our previous study despite the fact that, in this study, there is one activity more and activities are recognized body position-independently. However, the results from these two studies are not fully comparable because of different data sets. In additon, hardware variations between phones is not an issue, this can be seen from the results because the training data is collected using five different smartphones and activities from each phone are detected reliably. To show that hardware variations are not an issue in real-time on device recognition either, activity recognition applications for Symbianˆ3- and Android- operating systems based on the models used in offline tests were also coded. B. Online recognition An activity recognition application for Symbianˆ3 devices was built using Qt [13] programming language. In addition, the Android-version of the activity recognition application was built using Java programming language and tested using Samsung Galaxy Mini smartphone. The purpose of the online recognition experiment is to show that the presented method is not dependent on hardware and operating system. 1) Results: Online recognition on the device was tested by a person carrying several devices: five Nokia N8 smartphones and Samsung Galaxy Mini. He carried these phones at each of the five body locations. The recognition rates of different activities and recognition rates of activities from different body positions using Nokia N8 are presented in Table IV and the ones using Samsung Galaxy Mini in Table V. 2) Discussion: Online recognition accuracies are high, however, not as high as offline recognition results. This is understandable because it is impossible to reckon with all the situations of every day life. Now, for instance, the recognition rates for driving a car in the online case were not as high as in the model training phase, bacause there were a lot of red traffic lights. These stops were classified as sitting instead of driving a car, and thus, making the detection rate for drivingactivity the lowest. In addition, while cycling, people tend to just taxi at times, so they do not pedal constantly. Therefore, it is a matter of an opinion if one concludes these events as cycling or not. This sort of situation was tested and it seems that these events are classified as driving a car. This is logical because both events cause quite similar vibrations due to the roughness of the road. Real-time recognition was tested using Android- and Symbianˆ3- phones. According to the results presented in

5 TABLE II OFFLINE ACTIVITY RECOGNITION RESULTS USING QDA. True class / predicted sitting/standing walking cycling running on table driving class sitting/standing 92.0% 0.1 % 0.0 % 0.0 % 3.0 % 4.9 % walking 0.0 % 98.0% 1.7% 0.3 % 0.0 % 0.0 % cycling 0.0 % 1.3 % 98.7% 0.0 % 0.0 % 0.0 % running 0.0 % 0.0 % 0.0 % 100.0% 0.0 % 0.0 % on table 1.3 % 0.0 % 0.0 % 0.0 % 98.7% 0.0 % driving 0.0 % 0.0 % 0.0 % 0.0 % 0.0 % 100.0% TABLE IV REAL-TIME ON DEVICE RECOGNITION RATES OF ACTIVITIES AND THE RECOGNITION RATES OF ACTIVITIES FROM DIFFERENT BODY POSITIONS USING SYMBIANˆ3 SMARTPHONE. Activity Recognition rate Body position Recognition rate walking 94.5 % trousers 94.2 % cycling 93.5 % jacket 94.9 % running % backpack 92.9 % sitting/standing 90.8 % ear % driving 84.5% brachium 93.1 % on the table 99.2% table 99.2 % TABLE V REAL-TIME ON DEVICE RECOGNITION RATES OF DIFFERENT ACTIVITIES AND THE RECOGNITION RATES OF ACTIVITIES FROM DIFFERENT BODY POSITIONS USING ANDROID SMARTPHONE. Activity Recognition rate Body position Recognition rate walking 94.4 % trousers 90.8 % cycling 91.8 % jacket 90.2 % running 95.4 % backpack 96.3 % sitting/standing 89.0 % ear 100.0% driving 85.1% brachium 90.5 % on the table 93.2% table 93.2 % Tables IV and V, there is not much difference in the recognition rates between operating systems. Neither when the rates between activities are compared, nor when the rates between body positions are compared. In fact, in both cases, the difference is not statistically significant according to the paired t-test. However, sitting/standing -activity was an exception. Both operating systems recognized it with approximately same accuracy but misclassification were classified to different classes. Symbian misclassified it to on table -activity while Android misclassified it as driving. It should be further study what caused this difference. However, the results are not fully comparable because not all phones were carried at the same time. Still, it seems that different hardware and operating system do not affect heavily the detection rates. However, in can be said that the presented activity recognition system is hardware and operating system independent. It works with every smartphone where the maximum sampling rate of the acceleration sensors is higher than 40Hz. It is light, too, Symbianˆ3 -version of the application uses around 15% of CPU capacity of the Nokia N8; thus, other applications can be run alongside. V. CONCLUSIONS This study presented an ready-to-use activity recognition system, it does not require any calibration or user inputs. Offline experiments were made using a data set collected from seven subjects. The experiment showed that the presented method is user-independent, activities from every subject were recognized with a high accuracy. In addition, this experiment showed that activities can be classified reliably regardless of the phone position and orientation. What is more, the online experiments done using six smartphones running two different operating systems, and because there is no significant difference between detection accuracies between operating systems, the recognition system is not dependent on the phone hardware or operating system because of the calibration independent features. Also, the different maximum sampling rates of different phones are not an issue, because in this study, the maximum sampling rate was not used. Although the accelerometers were running at full speed, a new value to be used in activity detection was nevertheless called at every 25 millisecond. Therefore, the used frequency was 40Hz, which is much less than the maximum frequency of most of the smartphones. Thus, the presented method can be used with every smartphone and it is not dependent on the phone model. While in the offline scanerio activities are classified almost perfectly, the real-time on device recognition results are not quite as high. However, also the online recognition rates are high, over 90%, with both tested operating systems. The difference between recognition rates of online and offline scenarios

6 is most likely caused by real-life situations from which there is no training data. Also other researchers have noted that reallife recognition rates are often lower than offline accuracies. In [14], it is studied how the recognition rate decreases when activity recognition models trained using the supervised data are tested with unsupervised data. In the study, the model was trained to recognize nine different sports/everyday activities and when it was tested using supervised data, the recognition rate was 91%. When the same model was tested using unsupervised data, including four out of nine trained activities, the average recognition rate was only 64%. When these recognition rates are compared to the results of this study, it can be noted that online recognition works quite well. Moreover, the detection accuracies can be further improved by building a behavior recognition system based on the activity classification results. Although the purpose of the real-time on device experiment was to show that activity recognition works regardless of the hardware and operating system differences, in future studies, it should be tested by a larger amount of test subjects. Now only one test person was used. In addition, it would be beneficial to collect more training data as well. This could reduce the differences between online and offline recognition rates. Moreover,null-data recognition is not included in this study, either, and therefore, such activities cause incorrect classifications. Thus, to improve the accuracy of the application, the nullactivity recognition should be included. In this study, everything except model training is done on the device. Another option would be to send the sensor data to the server, perform the classification process there and, finally, send the results to a smartphone using 3G. In this case, calculation capacity would not be an issue, but on the other hand, privacy issues should be handled. In addition, data transfer is not free and can cause exceptionally high costs, especially abroad. [4] D. Peebles, H. Lu, N. D. Lane, T. Choudhury, and A. T. Campbell, Community-guided learning: Exploiting mobile sensor users to model human behavior, in Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, [5] J. Ryder, B. Longstaff, S. Reddy, and D. Estrin, Ambulation: A tool for monitoring mobility patterns over time using mobile phones, in Computational Science and Engineering, CSE 09. International Conference on, vol. 4, aug. 2009, pp [6] L. Sun, D. Zhang, B. Li, B. Guo, and S. Li, Activity recognition on an accelerometer embedded mobile phone with varying positions and orientations, in Ubiquitous Intelligence and Computing, ser. Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2010, vol. 6406, pp [7] S. Wang, C. Chen, and J. Ma, Accelerometer based transportation mode recognition on mobile phones, in Wearable Computing Systems (APWCS), 2010 Asia-Pacific Conference on, april 2010, pp [8] J. Frank, S. Mannor, and D. Precup, Activity recognition with mobile phones, in Machine Learning and Knowledge Discovery in Databases, ser. Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2011, vol. 6913, pp [9] J. R. Kwapisz, G. M. Weiss, and S. A. Moore, Activity recognition using cell phone accelerometers, SIGKDD Explor. Newsl., vol. 12, pp , March [10] Nokia N8, [11] D. J. Hand, H. Mannila, and P. Smyth, Principles of data mining. Cambridge, MA, USA: MIT Press, [12] E. Haapalainen, P. Laurinen, H. Junno, L. Tuovinen, and J. Röning, Feature selection for identification of spot welding processes, Proceedings of the 3rd International Conference on Informatics in Control, Automation and Robotics, pp , [13] Qt, [14] M. Ermes, J. Pärkkä, J. Mäntyjärvi, and I. Korhonen, Detection of daily activities and sports with wearable sensors in controlled and uncontrolled conditions, Information Technology in Biomedicine, IEEE Transactions on, vol. 12, no. 1, pp , Jan ACKNOWLEDGMENT This work was done as a part of MOPO study (Clinical- Trials.gov Identifier: NCT ). The authors would like to thank Infotech Oulu and the Finnish Funding Agency for Technology and Innovation for funding this work. Special thanks to Tero Vallius who helped with the coding work and to Johanna Pyy for collecting the data. REFERENCES [1] P. Siirtola and J. Röning, Recognizing human activities userindependently on smartphones based on accelerometer data, International Journal of Interactive Multimedia and Artificial Intelligence, vol. 1, no. 5, pp , 06/ [2] H. Lu, J. Yang, Z. Liu, N. D. Lane, T. Choudhury, and A. T. Campbell, The Jigsaw continuous sensing engine for mobile phone applications, in Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems, ser. SenSys 10, 2010, pp [3] T. Brezmes, J.-L. Gorricho, and J. Cotrina, Activity recognition from accelerometer data on a mobile phone, in Distributed Computing, Artificial Intelligence, Bioinformatics, Soft Computing, and Ambient Assisted Living, ser. Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2009, vol. 5518, pp