Mobile and Ubiquitous Computing: Mobile Sensing Master studies, Winter 2015/2016 Dr Veljko Pejović Veljko.Pejovic@fri.uni-lj.si Based on: Mobile and Ubiquitous Computing Mirco Musolesi, University of Birmingham, UK
Mobile Phone Sensing Environment sensing Fixed indoor sensors Specialised mobile sensing solutions (early 2000s): Sociometer MSP
Mobile Phone Sensing Phone manufacturers never intended their devices to act as general purpose sensing devices Sensing components used to improve the interaction with the phone: Accelerometer to trigger screen rotation Gyroscope for playing games Microphone for making calls Camera for taking conventional photos
Mobile Phone Sensing Phone sensing requires a significant engineering effort: Frequent sampling with what was supposed to be an occasionally used feature Accuracy problems Battery lifetime Processing overhead Android is trying to lower the sensing overhead: E.g. Google Play Services for location updates Manufacturers start viewing sensors as a central component of their platforms
Infrastructure vs Phone Sensing Infrastructure sensor networks (WSNs) Well suited for sensing the environment Specialized hardware accurately monitors specific phenomena All resources dedicated to sensing High cost of deployment and maintenance (regular recharging thousands of sensor nodes) Phone sensing Personalised suited for sensing human activities General purpose hardware, often inaccurate sensing of the target phenomena Multi-tasking OS. Main purpose of the device is to support other applications Low cost of deployment and maintenance (millions of users where each user charges their own phone) Apps could get uninstalled
Smartphone Sensors Accelerometer Magnetometer GPS Light Camera Barometer Gyroscope Proximity Microphone WiFi Bluetooth GSM NFC Touch screen Thermometer Humidity sensor
Applications of Mobile Sensing Individual sensing: Fitness applications Behaviour intervention applications Group/community sensing: Sense common group activities and help achieving group goals, environmental sensing Urban-scale sensing: Large scale sensing - a large number of people have the same application installed; e.g. tracking speed of disease across the country Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles,Tanzeem Choudhury, Andrew T. Campbell.A Survey of Mobile Phone Sensing. IEEE CommunicaEons Magazine. September 2010.
Properties We Can Infer Physical activity (running, walking, sitting) Accelerometer Transport mode (bicycle, car, train) Accelerometer, GPS, WiFi Surroundings, context (party, shopping mall) Microphone, camera, Bluetooth Human voice (speaker recognition, stress) Microphone Many other things: Emotion, depression, sociability, etc.
Phone as a Societal Sensor
Reality Mining Study (2006) Phone is a wearable sensor allowing us to understand human behaviour 100 users at MIT campus carried a phone and ran an app sensing their GSM connectivity (towers), Bluetooth environment Data are available: realitycommons.media.mit.edu Find structure in individual and group behaviour
Reality Mining Study (2006) Association with GSM towers indicates the location, Bluetooth contacts indicate social encounters : Low-entropy subject (e.g. a staff member)
Reality Mining Study (2006) Association with GSM towers indicates the location, Bluetooth contacts indicate social encounters : High-entropy subject (e.g. a student)
You are entropic!
Reality Mining Study (2006) Other findings: Indoor we spend time close to static Bluetooth devices. Implication: do we need GPS for location inference? Bluetooth contacts can be used to infer the hierarchy of organisation Bluetooth contacts in time can be used to infer who is a friend and who is just an acquaintance
Reality Mining Study (2006) Issues: Privacy issues User compliance: Restart the app, forget the phone at home, forgets to recharge the phone Energy consumption versus sensed data granularity
CenceMe Study (2008) Infer a user s physical activity, and social environment using mobile phone sensors Small scale study: 22 people, three weeks Inferred properties: Activity Conversation Location Mobility mode Social context
CenceMe Study (2008) Get high-level inferences from low-level data: Sample low-level data; how often? Extract useful features accelerometer mean, variance, peaks Train a classifier with labelled ground truth data samples collected when we know whether a user is walking, sitting, running, etc. Classify decide the label for newly-seen data
CenceMe Study (2008) Recognise human conversation Feature selection: Apply FFT (Fast Fourier Transform) on the audio samples and compare training data Voiced Human voice contains most of its energy within Non-voiced a narrow range of frequencies
CenceMe Study (2008) Recognise human conversation Classification: Simple threshold on mean and std deviation of FFT-ed signal
Social fmri (2011) Monitoring a medium-size social group and use mobile phones to impact human behaviour Infer user s physical activity and social habits, and provide an incentive in order to make people more physically active