How Big Data is Changing User Engagement

Transcription

1 Strata Barcelona November 2014 How Big Data is Changing User Engagement Mounia Lalmas Yahoo Labs London

2 This talk User engagement Definitions Characteristics Metrics Case studies: Absence time Within sessions Across sessions

3 User engagement

4 What is user engagement? User engagement is a quality of the user experience that emphasizes the phenomena associated with wanting to use a technological resource longer and frequently (Attfield et al, 2011) self-report: happy, sad, enjoyment, physiology: gaze, body heat, mouse movement, analytics: click, upload, read, comment, share emotional, cognitive and behavioural connection that exists, at any point in time and over time, between a user and a technological resource

5 Why is it important to engage users? In today s wired world, users have enhanced expectations about their interactions with technology resulting in increased competition amongst the purveyors and designers of interactive systems. In addition to utilitarian factors, such as usability, we must consider the hedonic and experiential factors of interacting with technology, such as fun, fulfillment, play, and user engagement. (O Brien, Lalmas & Yom-Tov, 2014)

6 Why is it important to measure and interpret user engagement well? CTR

7 Characteristics of user engagement Focused attention Novelty Positive Affect Richness and control Aesthetics Reputation, trust and expectation Endurability Motivation, interests, incentives, and benefits (Attfield etal, 2011)

8 Large scale measurements analytics within-session engagement measures success in attracting user to the site. across-session engagement measured by observing user long-term value. within-session measures Dwell time Session duration Bounce rate Play time (video) Click through rate (CTR) Number of pages viewed Conversion rate Number of UCG (comments). Time between visits across-session measures Fraction of return visits Total view time per month (video) Lifetime value (number of actions) Number of sessions per unit of time Total usage time per unit of time Number of friends on site (social networks) Number of UCG (comments) Time between visits activity loyalty popularity

9 Two case studies: Absence time news facebook news news mail absence time within sessions across sessions news

10 Case study I Absence time within sessions

11 The context Online multitasking Browsing the old way 1min facebook news 2min news 1min news 3min news mail Dwell time during a visit on news site: 7min on average news site Browsing nowadays 1min news 2min facebook news 3min 1min news mail news Dwell time during a visit on news site: 2.33min on average (1min 3min 3min)

12 The dataset July M users 785M page views Categorization of the most frequent accessed sites 11 categories (e.g. news), 33 subcategories (e.g. news finance, news society) 760 sites from 70 countries/regions short sessions: average 3.01 distinct sites visited with revisitation rate 0.10 long sessions: average 9.62 distinct sites visited with revisitation rate 0.22

13 Time spent at each revisit decreasing attention increasing attention constant attention complex attention Proportion of total dwell time on site Percentage of navigation type % 50% 0% p-value = 0.09 m = mail sites social media sites news (finance) sites news (tech) sites p-value = 0.07 m = p-value = 0.79 m = i th visit on site i th visit on site i th visit on site i th visit on site Teleportation Hyperlinking Backpaging Four patterns: decreasing, increasing, constant and complex Backpaging increasingly used at each revisit Metrics: 1. AttShift shift of attention 2. AttRange fluctuation

14 Time spent between each visit Cumulative probability Absence time [min] news (finance) news (tech) social media mail 50% of sites are revisited after less than 1min interruption of a task? There are revisits after a long break returning to a site to perform a new task? 1 st 2 nd 3 rd visit absence time Metric: 1. CumAct increase with time spent between visits

15 Models of browsing behavior Clustering of sites using multitasking and standard engagement metrics: CumAct, AttShift, AttRange Visit, Session Five models C1: 172 sites C2: 108 sites C3: 156 sites C4: 74 sites C5: 166 sites Visit dt [min] Session dt [min] CumAct dt,3 AttShift dt,4 AttRange dt,4

16 One task during a session C1: 172 sites mail, maps, news, news (soc.) C2: 108 sites auctions, front page, shopping, dating High dwell time per visit and during entire session Users return to continue a task (short absence time) C1: attention is shifting to another site C2: attention is shifting slowly towards the site Visit dt [min] Session dt [min] CumAct dt,3 AttShift dt,4 AttRange dt,4

17 Several tasks during a session C3: 156 sites auctions, search, front page, shopping C4: 74 sites front page, search, download Users perform several tasks during a session No simple activity pattern C3: Dwell time per visit is low, but dwell time per session is high Visit dt [min] Session dt [min] CumAct dt,3 AttShift dt,4 AttRange dt,4

18 Sites with low activity C5: 166 sites service, download, blogging, news (soc.) Users do not spend a lot of time on these sites Time between visits is short Attention is shifting towards the site Visit dt [min] Session dt [min] CumAct dt,3 AttShift dt,4 AttRange dt,4

19 Browsing behavior can differ between sites of the same category C2: 108 sites auctions, front page, shopping, dating 0.75 C3: 156 sites auctions, search, front page, shopping 0.75 C2: users visit site once to perform their task C3: users visit site several times to perform task(s) Visit dt [min] Session dt [min] CumAct dt,3 AttShift dt,4 AttRange dt,4

20 The message absence time within sessions Online multitasking happens How it affects a site depend on the site Towards a taxonomy of online multitasking One main continuous task versus user loyalty New metrics that account for time spent between visits and at each visit Applications: story-focused reading & network of sites (Lehmann etal, 2013)

21 Case study II Absence time across sessions

22 The context search experience

23 The context search experience

24 Absence time and survival analysis Users (%) who read story 2 but did not come back after 10 hours story 1 story 2 story 3 story 4 story 5 story 6 story 7 story 8 story 9 Users (%) who did come back

25 The dataset Ranking function on Yahoo Answer Japan Two-weeks click data on Yahoo Answer Japan: search One millions users Six ranking functions 30 minute session boundary

26 Examples of search session metrics Number of clicks Click at given position Time to first click Skipping Abandonment rate Number of query reformulations

27 Absence time and number of clicks on search result page survival analysis: high hazard rate = short absence control = no click 5 clicks

28 Absence time search experience search session metrics absence time 1. No click means a bad user experience 2. Clicking lower in the ranking suggests more careful choice from the user 3. Clicks after the 5 th result reflect poorer user experience; users cannot find what they are looking for 4. Clicking at bottom is a sign of low quality overall ranking 5. Users finding their answers quickly (click sooner) return sooner to the search application 6. Returning to the same search result page is a worse user experience than reformulating the query

29 The message absence time across sessions Easy to implement and interpret short absence is a sign of loyalty important indication of user satisfaction with search portal Can compare many things in one go No need to estimate baselines But need lots of data to account for noise Applications: direct display & churn rate (Dupret & Lalmas, 2013)

30 Conclusions

31 Online sites differ concerning their engagement (Lehmann etal, 2012) Games Users spend much time per visit Search Users come frequently and do not stay long Social media Users come frequently and stay long Niche Users come on average once a week e.g. weekly post Service Users visit site, when needed, e.g. to renew subscription News Users come periodically, e.g. morning and evening

32 Some final words If you cannot measure it, you cannot improve it William Thomson (Lord Kelvin) You cannot control what you cannot measure DeMarco The way you measure is more important than what you measure Art Gust