Avoiding the Hang ups - Understanding Mobile Phone Self-completion Surveys

Size: px

Start display at page:

Download "Avoiding the Hang ups - Understanding Mobile Phone Self-completion Surveys"

Franklin Horn
5 years ago
Views:

1 Avoiding the Hang ups - Understanding Mobile Phone Self-completion Surveys CASRO Technology Conference, New York, Presented by: AJ Johnson, Vice President, Global Operations, Ipsos

Republic Dominican Republic Egypt Ecuador France Germany Guatemala Hong Kong Hungary Ireland Italy India Iraq Japan Jordan Korea KSA Kuwait Lebanon Mexico Norway Panama Peru Philippines Poland

2 Ipsos around the world present in 64 countries Fast Facts Founded: 1975 Head Office: Paris Total revenues in 2007: 927.2m Number of countries with Ipsos offices: 64 Total number of countries we do research in: 100+ Argentina Australia Bahrain Belgium Bolivia Brazil Canada Chile China Colombia Costa Rica Czech Republic Dominican Republic Egypt Ecuador France Germany Guatemala Hong Kong Hungary Ireland Italy India Iraq Japan Jordan Korea KSA Kuwait Lebanon Mexico Norway Panama Peru Philippines Poland Portugal Puerto Rico Romania Russia Singapore Slovakia South Africa Spain Sweden Switzerland Syria Taiwan Thailand Turkey U.A.E U.K. Ukraine U.S.A Venezuela Number of full-time Ipsos employees: 8,800 Number of clients around the world: 5,000+ Interviews conducted each year: > 10 million

3 Introducing the experiment To test the effects of RECENCY, SURVEY PLATFORM and ENHANCED INVITATION on survey engagement Pre-recruited from the Ipsos Mobile Panel based on intent to watch a number of prime time shows on the main UK commercial TV channel Those very likely to watch were placed in to one of six experiment groups Online Mobile Enhanced invite Regular invite Enhanced invite Regular invite Next day Next day After show Next day After show Next day TV shows chosen for the experiment were Emmerdale, Coronation Street, Harry Hill s TV Burp and Dancing on Ice screened Jan 29 Feb We used Turf analysis to gain maximum TV show reach. Three questions (factual and non-factual) were asked via mobile or online to those who confirmed they watched the show Five minute follow-up survey sent to all who completed the experiment

Choosing the technology Mobile Internet Still only 12% use mobile internet in

participant Different phones, different browsers Mobile Application Permission

Ipsos MORI Tech Tracker, Feb 2009 Please Note: SMS is a clumsy survey

4 Choosing the technology Mobile Internet Still only 12% use mobile internet in UK Reliant on mobile internet connection at all times Issues around cost to participant Different phones, different browsers Mobile Application Permission needed to download application to each participant Technical issues around download process SMS 70% of the UK population use SMS The demographic profile of SMS users is more representative than Mobile internet users It s very easy to use and people respond quickly! UK allows free texts so participant pays nothing to take part in the survey Ipsos MORI Tech Tracker, Feb 2009 Please Note: SMS is a clumsy survey technology with limitations and will not be the future of mobile research! But, for now, it works well for short surveys amongst the UK general population

5 The Hypotheses 1. Mobile survey respondents' level of engagement is higher when the survey is administered at the time of the event (Time of survey) 2. Mobile survey respondents' level of engagement is higher than those of online survey respondents (Survey platform) 3. Mobile survey respondents' level of engagement is higher when an enhanced invitation is administered (Type of invitation) Also looking to discover: Do mobile participants respond faster? Does the quality of survey responses increase With the use of mobile technology versus online? When invitations are sent straight after the TV show versus next day? When we enhance the survey invite? Whether participants reported any technical problems Satisfaction level with aspects of mobile survey completion Preferred method for survey completion Potential for different types of mobile research in the future

6 Attempting to trigger and measure engagement "Engagement" as a psychological phenomenon occurs when the mind is highly focused on a specific task or activity. Tasks or activities that the mind finds interesting, important, fun or challenging are more apt to create a state of "focus" or "presence William H. MacElroy, President, Socratic Technologies, Inc. How we attempted to trigger engagement Use of new and different mobile data collection platform over standard online Sending survey at a time when they have focused on the TV show Enhancing survey invitation using a range of engaging techniques How we attempted to measure engagement Response / Screening / Dropout rates Length of time taken to complete survey Accuracy of results to factual questions Length of response to verbatim questions Stated intent to take more surveys / recommend Increased agreement that survey is interesting, innovative, and different Please note: Our original assumption regarding engagement and interview length of SMS surveys proved incorrect

7 The effect survey length has on quality Number of adverts recalled Not significant Length of verbatim Not significant Significant 300 Significant Online - No advert recalled Online - More than one advert recalled Mobile - No advert recalled Mobile - More than one advert recalled 0 Online - Verbatim under 40 characters Online - Verbatim > 40 characters Mobile - Verbatim under 40 characters Mobile - Verbatim > 40 characters Average time (secs) Average time (secs) Accuracy of plot question For participants who completed the experiment via mobile we found no clear association between completion time and these three measures of quality Significant Online - Plot not accurate Online - Plot accurate Not significant Mobile - Plot not accurate Mobile - Plot accurate Average time (secs)

8 Mobile (same day) versus Mobile (next day) Same Day Next Day , % 76.2% 10 0 % Recommend Response rate Screen out rate Dropout rate Take part Innovative Different Interesting Top Box Score % Statistically significant Same Day Next Day

9 Enhancing the mobile invite Normal Ipsos Media Survey - Dear Panellist - Your survey will follow. Thank you Enhanced Ipsos Media Survey - Dear <[Name]>, One of a select group, your opinion is valuable 3 Replies are FREE; 4 Earn 50 points 5on completion. Q1 (Max 4) 6 follows. Thanks 1. Personal salutation 2. Making the participant feel that they are special 3. Stressing the importance of their contribution 4. Make sure participant understands completion will not cost them 5. Confirm incentive fee 6. Confirm maximum length of survey 1 2

10 Mobile (Enhanced) versus Mobile (Non-enhanced) Enhanced Not enhanced , % 79.9% % Recommend Response rate Screen out rate Dropout rate Take part Innovative Statistically significant Different Interesting Top Box Score % Enhanced Not enhanced

11 Mobile (Next day) versus Online (Next day) Mobile 1, Online 76.2% 71.8% % 80% 70% 60% 50% 40% 30% 20% 10% 0% % Recommend Response rate Take part Innovative Different Interesting Screen out rate Dropout rate Top Box Score % Statistically significant Mobile Online

participant Quality Index + + Female Male 50% 51% 52% 53% 54% 55% 56%

12 Building a Quality Index By combining Accuracy, Verbatim length, and Advertisement recall we created a quality index score for each participant Quality Index + + Female Male 50% 51% 52% 53% 54% 55% 56% 57% Quality Index scores are significantly higher amongst female participants

13 Quality Index Mobile versus Online 60.0% 58.0% 56.0% 54.0% 52.0% 50.0% 48.0% Quality Index scores are significantly higher from mobile responders versus online responders 46.0% Mobile Next Day Online Next Day Quality Index scores were higher for MOBILE SAME DAY v MOBILE NEXT DAY and MOBILE ENHANCED v MOBILE NOT ENHANCED, but not significant

14 Average Quality Index across the designs Our expectation was Online Not Enhanced Online Enhanced Mobile Not Enhanced Next Day Mobile Enhanced Next Day Mobile Not Enhanced Same Day Mobile Enhanced Same Day 62% 60% 59.87% 60.21% 60.35% 58% 56% 56.85% 54% 52% 50.55% 50% 51.27% 48% 46% 44% Online, Not Enhanced, Next Day Online, Enhanced, Next Day M obile, Not Enhanced, Next Day M obile, Enhanced, Next Day M obile, Not Enhanced, Same Day M obile, Enhanced, Same Day

15 Response rate (%) Survey response times Online versus Mobile % of mobile survey responses in the first two hours Overall higher response rate from online survey Online Mobile Time (hours) Key Finding: Mobile participants respond much faster but response rates are lower

16 Any technical problems? Online Mobile Total Yes No Don't know / Can t remember I wasn t shown adverts No mobile reception Phone jammed Survey froze Battery ran out Didn t receive the whole survey

17 Satisfaction with SMS completion Ease of question completion 77.2 Speed of text delivey 78.5 Ability to read text % Top Box (Very satisfied) Key Finding: High satisfaction levels with SMS survey completion

18 Preferred method of survey completion 100% 90% 88.9% 80% 70% 60% 50% 40% 30% 20% 10% 0% 17.1% 8.6% 1.1% 33.7% 63.0% 49.3% 28.4% 10.0% Via mobile phone Via the internet using a computer No preference Completed Online Completed via SMS Total Key Finding: The conversion effect Those who complete via mobile are more likely to state mobile as preferred completion method

19 Future mobile research intent Location based surveys Completing diaries Sending vidoes Sending phots WAP Surveys (Via mobile internet) Text Surveys (SMS) % Top Two Box (Very likely, Fairly likely) Statistically significant Completed Online Completed via SMS Again we see the Conversion Effect

Happy participants The text surveys are a good idea and of course your mobile is always on so they can be completed very quickly You never miss a survey doing it this way SMS was an interesting

20 Happy participants The text surveys are a good idea and of course your mobile is always on so they can be completed very quickly You never miss a survey doing it this way SMS was an interesting addition to the survey spectrum I really enjoyed completing this survey. I thought it was an innovative way to be involved 72% Very enjoyable and easy to follow...this is the way ahead, using SMS you will get the answers quicker as opposed to it sitting in someone's inbox for however long...

Unhappy participants I enjoyed taking the survey by mobile telephone but It is much more expensive as I have to pay for each answer I send Receiving surveys by text was inconvenient due to me being

21 Unhappy participants I enjoyed taking the survey by mobile telephone but It is much more expensive as I have to pay for each answer I send Receiving surveys by text was inconvenient due to me being at work at the time It would be helpful to know how many text messages would be received before starting the survey as they just seemed to keep on coming I didn t like the SMS survey as it became tedious and because the texts kept coming I didn t enjoy it. I much prefer doing it online 28% Sometimes have problems with the reception so may not be able to complete the survey

22 Finally The outlook for research via mobile phones Mobile research - We're on the verge of the next big thing in market research Perhaps not, but the foundations are beginning to take shape! Mobile Research 09, London The market research industry cannot drive these developments but will benefit from those who do!

The opportunities of mobile access at Ipsos It is becoming increasingly difficult to access busy and affluent groups in the developed world.

The mobile phone offers an opportunity to reach people on the move in a markets where mobile devices out number people Young people are

Usage is moving away from voice and texting to social networking via their mobile devices.

23 The opportunities of mobile access at Ipsos It is becoming increasingly difficult to access busy and affluent groups in the developed world. The increase of mobile only households and DNC lists limits traditional access methods. The mobile phone offers an opportunity to reach people on the move in a markets where mobile devices out number people Young people are increasingly reliant on the use on mobile phones to communicate. Usage is moving away from voice and texting to social networking via their mobile devices. The developing world represents great potential for mobile. Mobile phone penetration surpasses fixed line and for many the mobile phone offers the single point of access to the internet. Almost all of the next 1 billion new mobile users will be from emerging nations. Source: Patrick Dixon, Futurist, Keynote MTN Conference, July 2008

24 The challenges of mobile engagement Like this? Will people who like This And this And this And this And this And this And this! The attention span of the consumer is ever decreasing in a world that offers an increasing number of distractions. To be successful we must carefully consider how we can engage our survey participants

25 Thank you How would you spend the next 5 minutes? Boring Survey Sexy Girl Contact: aj.johnson@ipsos.com

Access Code and Phone Number

Access Code and Phone Number Algeria Dial International collect/reverse charge number: 1-212-559-5842 Argentina For phones using Telecom: Dial 0-800-555-4288; wait for prompt, then dial 866- For phones using Telefonica: Dial 0-800-222-1288;