Tap Position Inference on Smart Phones

Transcription

1 Tap Position Inference on Smart Phones Ankush Chauhan Outline Introduction Application Architecture Sensor Event Data Sensor Data Collection App Demonstration Scalable Data Collection Pipeline Feature Extraction Pipeline Feature Engineering Screen Partitioning Data Visualizations Model Selection Comparing Accuracies Conclusion Links Related Works 1

2 Introduction Immersive Full Screen Data collection application for Tap Event Detection Scalable Data Collection Pipeline Sensor Fusion algorithm for extracting rotations Clustering of similar regions based on screen partitioning Ensemble Machine Learning Approach to combine multiple ML models. Interactive Notebooks for data analytics Application Architecture Recording Sensors No explicit manifest declaration required. CallBack Event Listener onsensorchanged() Service Records sensor data event in background Log data to a CSV file File name contains Experiment Epoch Each record contains lastsensorvalues() with Timestamp as event time delta from Epoch. Recording screen taps Full Immersive Layout Highest granularity of 20px X 20px region Tabular Arrangement 16 Rows 9 Columns Lowest granularity of 1920px X 1080px region Store valid taps in a defined JSON schema to NOSQL database. 2

3 Sensor Event Data Motion Sensors used for capturing sequences on 3- coordinate axis Accelerometer acceleration force measurements Magnetometer magnetic field measurements Gyroscope angular velocity measurements Delay SENSOR_DELAY_FASTEST (no delay) SENSOR_DELAY_GAME (20,000 microsecond delay) Sensor Data Collection 01 For complete duration of one experimental run, we log raw updates for selected sensors to a temporary cache file on device. 02 On leaving the app activity the cached file is uploaded to Firebase storage for staging the data. 3

4 TapLogger App Demo Record Sensor Event Log Training Ends Upload Log Cloud Storage TapLogger Launched Initialize Data Capturing Modules Tap Event Insert Event Realtime Database On-Exit Event Handler On-Click Event Handler 4

5 Load Sensor Recording CSV Parser Sensor Data Frame Transform Features Feature Set Sensor Data Staging Extracted Sensors Features + Target Labels Load Tap Recordings JSON Parser Tap Data Frame Transform Labels Label Set Tap Data Staging Rotations in 3D Roll Pitch Yaw 5

6 Feature Engineering Sensor data filtered using Madgwick AHRS algorithm for extracting rotations in phone s frame of reference. Decomposing Rotations from MAG; filter and update technique to extract Euler Angles (Roll, Pitch, Yaw) 3 rotations Cons Ambiguity, Gimbal Lock Quaternions 1 rotation, as Sin and Cos captured. (q0,q1,q2,q3) Cons - Human Interpretability For each such field, we calculate it s first four statistical moments as a feature set in the tap interval sequence. F [start end] = F Mean F Variance F Skewness F Kurtosis Corresponding to each tap event we divide the feature sequence to two monotonic sections. F Down and F Up 2 Target Regions 4 Target Regions 8 Target Regions 16 Target Regions 6

7 3 Target Regions 9 Target Regions 7

8 8

9 Horizontal Partitions 1200 Vertical Partitions Class wise Sampling Distribution

10 Feature Correlation Quaternions Euler Angles Model Selection OneVsRestClassifier LR - Logistic Regression CART Decision Tree NB Gaussian Naïve Bayes LDA Linear Discriminant Analysis KNN K Nearest Neighbors SVM Support Vector Machine Ensemble technique known as majority voting, which combines the predicted class for a particular class label that represents the majority (mode) of the class labels predicted by each individual classifier. 10

11 2 Class Classification Accuracy on Euler Angles 11

12 2 Class Classification Accuracy on Quaternions 12

13 3 Class Classification Accuracy on Euler Angles 13

14 3 Class Classification Accuracy on Quaternions 14

15 4 Class Classification Accuracy on Euler Angles 15

16 4 Class Classification Accuracy on Quaternions 16

17 Our learning system has been able to predict better than a random guess which screen area has been tapped. Validation on lower granularity will require combining correlated features. Conclusion Attack strategies can be evaluated for confirming model robustness. Suggested mitigation efforts include additive noise strategies to mask those micro-level changes in sensor reading. Data Science Stack Scientific Computation Machine Learning Data Structures Data Visualization Numerical Computation Interactive Kernel Notebooks 17

18 Project Repositories Xu, Zhi, Kun Bai, and Sencun Zhu. "Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors." Proceedings of the fifth ACM conference on Security and Privacy in Wireless and Mobile Networks. ACM, Zheng, Nan, et al. "You are how you touch: User verification on smartphones via tapping behaviors." Network Protocols (ICNP), 2014 IEEE 22nd International Conference on. IEEE, Related Works Madgwick, Sebastian OH, Andrew JL Harrison, and Ravi Vaidyanathan. "Estimation of IMU and MARG orientation using a gradient descent algorithm." Rehabilitation Robotics (ICORR), 2011 IEEE International Conference on. IEEE,

19 Questions? Thank You 19