Usability Testbed for Website Authentication Technologies

Transcription

1 Usability Testbed for Website Authentication Technologies Maritza Johnson, Chaitanya Atreya, Adam Aviv, Mariana Raykova, Bryan Gwin, and Steve Bellovin 1

2 Financial Services Technology Consortium: Authenticating the Financial Institution to the Customer 2

3 Overview Problem description Usability Testbed User study design Test harness Results Microsoft CardSpace Verisign Secure Letterhead 3

4 How does a user know what site they re on? 4

5 5

6 But Users Rely On 6

7 Phishing 7

8 Proposed Solutions New login procedures Anti-phishing toolbars Displaying additional information in the browser 8

9 Deploying New Technologies The implementation might be decent, but is it usable? 9

10 Defining Usability Spoofability Learnability Acceptability 10

11 Is a Testbed Necessary? Need a mechanism for evaluating technologies before they re deployed Need a design that takes into account each aspect of our usability definition Results gathered from different user study designs are seldom directly comparable 11

12 Evaluation After Deployment Text Text Schechter, Stuart E.; Dhamija, Rachna; Ozment, Andy; Fischer, Ian, "The Emperor s New Security Indicators," Security and Privacy, SP '07. IEEE Symposium on, vol., no., pp.51-65, May

13 Usability Testbed User Study Design Test Harness 13

14 User Study Design Must evaluate each aspect of our usability definition Must avoid creating an unrealistic focus on security Must have tasks on online banking sites without collecting sensitive personal data 14

15 Method of Evaluation In lab study Survey Tasks self-reported feedback between tasks Post study questionnaire 15

16 Evaluating Spoofability Trick the participant into going to spoofed sites Measure if the participant enters personal information on a spoofed site Study Design Send tasks by , make half of them phishing s Give the option to not complete a task 16

17 Evaluating Learnability Does the user notice the technology is in place? Is effective use possible without instruction? Do instructions contribute to better use? Study Design Separate the tasks by an instructional Compare number of successful spoofs before and after instructions 17

18 Evaluating Acceptability Does the user feel the task warrants the process? Does the user recognize value in the process? Study Design Questions to draw feedback in post-study questionnaire 18

19 Additional Design Questions Role to justify using someone else s account information Wording of the post-task and post-study questionnaires Describing the study purpose without focusing on security 19

20 Session Overview Consent Form Survey Instructions, role, account information 8 tasks alternating real and spoof (4/4) post-task questionnaire task 5 - instructions Post-study questionnaire

21 Test Harness Proxy to serve the webpages Real pages Spoofed pages script Beacons for logging page transitions s 21

22 Testing the Testbed Validate the process and show useful data can be collected with our methodology Demonstrate two different technologies can be tested 22

23 IRB Approval Institutional Review Board IRB training (~ 45 min) Human subjects protocol Consent Form Meetings every 2 weeks Must submit 2 weeks prior to meeting 23

24 Human Subjects Protocol Study purpose and design Participant recruiting Confidentiality and storage of data Potential risks and benefits 24

25 Pilot Two sessions Improved the websites used in the study Reworded some of the s and questions 25

26 Recruiting Posters around campus (4) Facebook Marketplace (1) Craigslist (13) 18 participants Cardspace (13) Secure Letterhead (5) 26

27 Microsoft CardSpace 27

28 CardSpace Participants 4 female, 9 male (age 18-60) All spend >20 hours/week on the Internet All but one use online banking Webmail: Gmail, Hotmail, Yahoo 1 victim of identity theft (IT) Browser: Firefox, IE, Safari, Opera Paypal, Wamu, Bank of America, Ebay, Chase 28

29 Without Instructions 12/13 commented they were confused in task 1 11/13 didn t notice anything suspicious recognized the spoof after submitting info 11/13 fell for the second spoof recognized the spoof without entering info IT only used the URL to make their decision 6 commented it was intuitive in the post-study Q 29

30 After Instructions 2 were still confused in the task that followed 3/13 didn t fall for the first spoof 2 cited the instructions 3/13 didn t fall for the second spoof 1 cited the instructions 1 remarked the site just looked fake One who didn t fall for the previous spoof fell for the second one 30

31 Post-Study Feedback The amount of time required 4 - Slightly too much 6 - perfect amount The information provided: 6 - is useful and seems necessary 5 - appears useful but might not be necessary 2 - is not useful at all 31

32 Secure Letterhead 32

33 Simple Spoof 33

34 Secure Letterhead Participants 2 female, 3 male (age 18-50) Webmail: Gmail, Hotmail Browser: Firefox, Safari, IE All spend >20 hours/week on the Internet ING, Amazon, Paypal, Wamu, Amex, Ebay 1 person was knowledgeable about phishing 34

35 Without Instructions The first spoof fooled all 5 participants One participant realized they were spoofed and wasn t spoofed again, but Secure Letterhead did not play a part in their decision Prior to the instructions, nobody noticed Secure Letterhead in the chrome and no one attempted to access the pop up for more info 4/5 fell for the second spoof 35

36 After Instructions 5/5 attempted to check for it in the task immediately following One commented the instructions were incorrect and the logo was on the left The previously phished didn t complete the task and stated they didn t trust the information One stated the was too wordy, then tried to click the logo in the page content In the remaining 3 tasks, the information was accessed on 2 occasions, once was after a user had been redirected by the phishing page to the real page 36

37 Post-Study Feedback Curious about how easy it would be to fabricate 3/5 reported they would remember to check for the logotype before doing something important 2/5 stated they wouldn t have figured out how to use it without instructions Information doesn t seem useful Information seems useful, but maybe not necessary. 37

38 Testbed Strengths Our testbed can be used for a range of website authentication technologies with only slight modifications. Comparisons between technologies can be done easily with the data collected. Larger studies can be ran to collect a statistically significant amount of data. 38

39 Testbed Weaknesses Asking the participants to play a role may lead to them acting less secure Self reported data is not always reliable Direct observation can yield more insight Effect of the lab setting on results is unknown Not able to evaluate performance over time 39