How well can navigation research with different design artifacts predict final site performance?

Transcription

1 How well can navigation research with different design artifacts predict final site performance? IA Summit 2016 David Juhlin Bentley University User Experience Center May 7, 2016

2 About me 2

3 Thanks 3 Bill Albert Dani Norden

4 Setting the Stage 4

5 Setting the Stage 5 Which methods/artifacts, when? How do you interpret data? What does it say about the final design?

6 Motivation 6 How predictive is early UX research on a final design? Help organizations avoid bad decisions Sky is falling might lead to radical and unnecessary changes We re good to go might lead to lack of improvements

7 Visual treatment Research Question 7 High Design comps Wireframes Mid-fidelity prototype Production ready prototype High-fidelity prototype Final live site Low Sketch IA Tree Proof of concept prototype Low Interactivity High

8 Visual treatment Research Question 8 High Design comps Wireframes Mid-fidelity prototype Production ready prototype High-fidelity prototype Final live site Low Sketch IA Tree Proof of concept prototype Low Interactivity High

9 Design is not a straight path 9

10 All design is not as winding 10

11 A Few Details 11 Wells Fargo s website (Fall, 2014) Reversed engineered the live website into each artifact (high-fidelity prototype, design comp, wireframe, IA tree) UserZoom to collect data (success, time, unique pages, clicks) Participants recruited through Amazon s Mechanical Turk ( participants for each artifact) 7 common tasks Analyzed correlations between each artifact and the live site

12 Live Website ( 12 Metrics: success, time, unique pages, and clicks Unmoderated usability test (n=100)

13 Tasks 13 T1 - Overdraft T2 - Locations T3 - Interest rate T4 - Insurance T5 - ATM fee T6 - Internship T7 - Retirement Task How much is an overdraft if you have linked your checking account to your savings account? How far away is the closest Wells Fargo branch? What is the interest rate for a Way2Save savings account? What is covered if you purchase insurance for your apartment? What is the ATM fee for a withdrawal in Mexico? What majors are desired if you want to apply for an internship in the audit division? What is the recommended amount you should be placing in your 401k if you are 22 years old?

14 High-Fidelity Prototype 14 Black/white Simplified images Consistent terminology and layout 64 pages No Small Business or Commercial Unmoderated usability test (n=99)

15 Live site success rate High-Fidelity Prototype: Task Success 15 Success on the high-fidelity prototype reliably predicts success on the live site (r= 0.961, p<.001) % 90.00% 80.00% 70.00% Success y = x Prototype success rate Live site predicted success rate 20% 32% 30% 40% 40% 48% 50% 56% 60% 65% 70% 73% 80% 81% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% 0.00% 20.00% 40.00% 60.00% 80.00% % Prototype success rate

16 Live site Mean Time High-Fidelity Prototype: Completion Time 16 Completion time on the highfidelity prototype is an excellent predictor of completion time on the live site (r=0.960, p<.001) Time y = x Prototype Prediction of Time (s) Live site Prototype Mean Time

17 Live site Mean Unique Pages High-Fidelity Prototype: Unique page views 17 The number of unique page views on the high-fidelity prototype was a good predictor of unique page views on the live site (r=0.943, p<.001) Prototype Prediction of Unique Pages Live site Unique Pages y = x Prototype Mean Unique Pages

18 Live site Number of Clicks High-Fidelity Prototype: Clicks 18 The number of clicks on the highfidelity prototype is a good predictor of the number of clicks on the live site (r=0.814, p=.026) Number of clicks is understated in the prototypes for lower values Prototype Prediction of Clicks Live site Clicks y = x Prototype Number of Clicks

19 Prototype: Summary 19 Live Site Prototypes- Success Prototypes- Time Prototypes- Pages Prototypes- Clicks Success Excellent r= p = Time Good r= p = Excellent r= p = Unique Pages Good r= p = Excellent r= p = Clicks Good r= p = Good r= p = 0.026

20 Prototype: Prediction % 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Success rate Prototype Live site

21 Design Comp 21 Screen shot (homepage only) Removed Search, Contact Us, and Login First click test (n=105)

22 Design Comp: Summary 22 Live Site Design Comp: Click Success Design Comp: Time Design Comp: Unique Pages Design Comp: Clicks Task Success Excellent Correlation = p = Time Good Correlation = p = Unique Pages Excellent Correlation = p = Clicks Good Correlation = p = 0.059

23 Design Comp: Prediction % 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Success rate Design Comp Live site

24 Wireframes 24 Similar to high-fidelity prototype except for the following Search, Login and Contact US were removed Lorem Ipsum text in some places First click test (n=105)

25 Wireframe: Summary 25 Live Site Wireframe: Click Success Wireframe: Time Wireframe: Unique Pages Wireframe: Clicks Task Success Good Correlation = p = Time Good Correlation = p = Unique Pages Excellent Correlation = p < Clicks Good Correlation = p = 0.015

26 Wireframe: Prediction % 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Success rate Wireframe Live site

27 IA Tree 27 Converted the mega menu into IA tree Primary navigation Some tasks could be solved without the mega menu Tree test (n=195)

28 IA Tree: Summary 28 Live Site IA Tree: Target Success IA Tree: Time IA Tree: Unique Pages IA Tree: Clicks Task Success Poor Correlation = p = Time Good Correlation = p = Fair Correlation =0.611 p=0.145 Unique Pages Fair Correlation = p = Clicks Fair Correlation = p = 0.088

29 IA Tree: Prediction % 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Success rate Tree test Live site

30 Overall Summary 30 High-Fidelity Prototype Screenshot IA Tree Design Comp Wireframe Live Site Task Success Time Pages Clicks Click Success Click success Success Time Task Success Excellent Excellent Good Poor Time Good Excellent Good Good Good Fair Unique Pages Good Excellent Excellent Excellent Fair Clicks Good Good Good Good Fair

31 Visual treatment Research Question 31 High Design comps Good/Excellent Predictor Final live site High-fidelity prototype Excellent Predictor Wireframes Good Predictor Low IA Tree Fair Predictor Low Interactivity High

32 Key Takeaways 32 Early UX research results does predict final site performance for more linear design projects. The closer the artifact is to the live site, the better the prediction. The farther the artifact is from the live site, the worse the prediction. Don t use one type of metric to predict another type of metric (e.g., success as a predictor of time). Limitations of the study include only a sample of artifacts and data, limited number of tasks, and a single website.

33 Implications 33 During formative research, focus on identifying issues, making improvements, and measurement If management are concerned about the results from early research don t let them read too much into it it s a long road to great design Promote UX validation research

34 Thank you! David Juhlin User Experience Bentley Univ. User Experience Center Linked in group- Bentley UXC