Feel the touch. touchscreen interfaces for visually impaired users. 21 November 2016, Pisa. 617AA Tecnologie assistive per la didattica

Size: px

Start display at page:

Download "Feel the touch. touchscreen interfaces for visually impaired users. 21 November 2016, Pisa. 617AA Tecnologie assistive per la didattica"

Hester Allen
6 years ago
Views:

1 Feel the touch touchscreen interfaces for visually impaired users 617AA Tecnologie assistive per la didattica 21 November 2016, Pisa

3 Text Entry on Touchscreen Mobiles Difficult for blind users due to lack of physical keys and layout references External keyboards (Qwerty or Braille) are not easily portable nor cheap Use of Automatic Speech Recognition (ASR) and Text-To-Speech (TTS) Inconvenient in noisy environments Privacy and etiquette concerns

4 Barbara Leporini

5 Barbara Leporini Blind since childhood President of the Italian Blind Association in Tuscany PhD and researcher on Human Computer Interaction and accessible interfaces Uses digital technologies on a daily basis

6 Barbara s cellphones Nokia N95 iphone 4S No touchscreen Physical keypad TALKS screen reader Symbian OS Touchscreen No physical keyboard VoiceOver screen reader ios 7

7 Why two cellphones? Nokia N95 Phone calls Text messages (SMS) Confortable audio feedback without earphones Longer battery life (no Internet access) iphone 4S Information and entertainment ebooks and news Music and podcast listening and online social networks

8 Text Entry on the iphone 4S Mainly via the virtual keyboard Lack of precision because of the small keys Trouble with adjacent letters with similar sounds (e.g., M and N) on noisy environments Difficult text navigation and revision Complicated editing at the word or phrase level Siri s ASR is used when alone and in quite environments only

9 Accessible Mobile Text Entry Software-only approaches TalkBack (Android) and VoiceOver (ios) Hardware-only approaches Plastic overlays (e.g., keyboard, maps) Hybrid solutions Plastic overlays with NFC chips

10 Software-only Approaches Multitouch Character Encoding BrailleTouch and DigiTaps Character Drawing Graffiti and ios Handwriting Constructive Methods Virtual keyboard with TTS

11 Our Approach: MTITK Multimodal Text Input Touchscreen Keypad Software-only approach based on previous work on haptic differentiation of UI elements Use of the common 12-button telephone keypad layout Use of gestures for additional actions Simultaneous voice and haptic feedback

12 Haptic Keytap Prototype An Android Input Method

13 Telephone Keypad ITU E.161 ISO/IEC ETSI ES

14 Text Editor Input Area Global Actions

15 Input Method (press gesture) 2 ABC 2 ABC Normal State Selected State (initial haptic feedback)

16 2 ABC press gesture + 1tap A 2 ABC press gesture + 2 taps B 2 ABC press gesture + 3 taps C

17 Backspace Read last char

18 Shift Down Shift Up

19 Block Shift Down Block Shift Up

20 Keyboard Layout

21 1 tap * 0 # * 0 # Shift down Shift up

22 1 a d - A D g j m G J M p t w P T W ( ) / \ Shift down Shift up

23 2 taps ' b e _ B E h k n H K N q u x Q U X.? [ + ] Shift down Shift up

24 3 taps " c f : C F i l o I L O r v y R V Y,? $ Shift down Shift up

25 4 taps & 2 3 ; s 8 z S 8 Z < = > ^ % Shift down Shift up

26 5 taps { 0 } ~ 0 Shift down Shift up

27 6 taps * # Shift down Shift up

28 Localization Keypad Layout and default Text To Speech engine US English (en_us) android default Spanish of Spain (es_es) French of France (fr_fr) Italian of Italy (it_it)

29 KEYPAD LAYOUT AND KEY GROUPS * #

30 KEYPAD LAYOUT AND KEY GROUPS Intercardinal point * #

31 KEYPAD LAYOUT AND KEY GROUPS Cardinal point * 0 #

32 KEYPAD LAYOUT AND KEY GROUPS Center * 0 #

33 KEYPAD LAYOUT AND KEY GROUPS Zero * 0 #

34 KEYPAD LAYOUT AND KEY GROUPS Symbol * 0 #

35 KEYPAD LAYOUT AND KEY GROUPS Intercardinal point Cardinal point Center Zero Symbol * 0 #

36 MTITK PROTOTYPE DEMO on Nexus 4 (Android 4.4)

37 SIMPLE TEXT ENTRY

38 REVIEW GESTURES Read / Delete : last character last word last sentence all text

39 MULTI-LANGUAGE ENGLISH FRENCH ITALIAN SPANISH

40 Pilot Test 14 participants: 6 females, 8 males, median age of 44 Self-reported most difficult text entry and editing tasks on the Qwerty virtual keyboard: autocomplete, select text and delete one or more words Difficulties to remember the keypad s character mapping despite having declared familiarity Some participants could not accurately differentiate the key groups vibration patterns Difficulties with right-angle gestures

41 Lessons and Observations (1) ios is the most popular among blind users More accessible features VoiceOver maturity Android is better suited for research Open nature and lower device cost But significant device fragmentation

42 Lessons and Observations (2) Some blind users don t know their devices Unaware of most accessibility features Knowledge transfer among blind users Research participation motivates blind users Some participants bought smartphones after experiencing current solutions

43 Lessons and Observations (3) Many participants had problems performing the prototype input gestures. Lack of knowledge on how blind users perform gestures on mobile phones touchscreens

44 How do visually impaired people perform touch gestures and what are their preferences?

45 Motivation Touch-based and screen reader interaction may be hampered by non-accessible design choices or the user s environment. No physical references Limited haptic feedback Noisy surroundings Privacy concerns Gesture performance

46 Participants 36 participants women 22 men Mean age: 48 years Std Dev: 15 years 12 Severe low vision Blind in adulthood Blind from adolescence Blind from birth 6 7

47 Participants use of touchscreen devices ios Android Other Group iphone ipod ipad Phone Tablet Phone MP3 Low-vision 45% 18% 55% 9% 18% Blind since birth 57% 14% 29% 14% 0% Blind since adolescence 83% 0% 17% 17% 0% Blind in adulthood 58% 25% 8% 0% 8% All of the participants 57% 17% 28% 8% 8%

48 Gesture Patterns Tap Swipe Pinch Letterlike shape Rotor Angled shape To and fro swipe 25 gesture patterns, 7 groups

49 Gesture Angle Curved Steep Right-angled

50 Gesture capture system

51 Nexus 5 Smartphone 69.2 mm 8.6 mm mm mm 61.6 mm

52 Conceptual model gesture data points JSON Gesture Preview Device A Device B Device C Participant1 Participant2 Participant3 WebSocket Connections JSON commands and parameters Web-based Server

53 Architecture Client devices device data Web server device data Web application Android Ruby / Sinatra NodeJS / Angular A settings WebSocket settings Model Participant Start Database participant data Controller Researcher SQLite gestures data View

54 Multi-Participant Participant 2 Participant 1 Participant 3 Gesture Preview Device A Device B Device C Participant 1 Participant 2 Participant Commands and Parameters Gesture Data Points 05 Capture Dashboard

55 How to explain gestures?

56 Analogies

57 Cardboard Cut-outs

58 Preferences

59 Perceived Gesture Difficulty 4,2 3,5 2,8 1,48 2,1 1,4 0,7 0 Swipe Left 2-Finger double tap Swipe Up 2-Finger Swipe R Swipe D-L Average Rotor CCW Swipe L-R-L (2F) Rotor CW 2-Finger letter Z Swipe U-D-U (3F) Easiest Average Most difficult

60 Perceived Gesture Difficulty 4,2 3,5 2,8 2,1 1,4 0,7 0 Swipe Left 2-Finger double tap 1,48 Longer gestures are perceived as more difficult Swipe Up 2-Finger Swipe R Swipe D-L Average Rotor CCW Swipe L-R-L (2F) Rotor CW 2-Finger letter Z Swipe U-D-U (3F) Easiest Average Most difficult

61 Gesture Preferences Shape Rounded No preference Steep

62 Gesture Preferences Number of Strokes At most one No preference At most two

63 Gesture Preferences Number of Fingers At most one No preference At most two At most three

64 Analysis examples

65 Two-finger double tap Right-handed Left-handed Blind since birth Blind since adolescence Became blind in adulthood Severe low vision

66 Severe low vision Blind in adulthood Blind f. adolescence Blind f. birth All groups

67 Invalid Captures Per Gesture Plus shape Circle Chevron U Swipe R-U Swipe Pinch Letterlike shape Tap Rotor CW Rotor CCW X shape Swipe D-U Pinch out Rotor Angled shape To and fro swipe Pinch in Swipe L-R-L Z shape 2F Swipe U-D-U 3F Swipe L-R-L 2F

68 Bounding Box Centers a) All bounding box centers b) Bounding box centers of the gesture Bracket U Blind Low-vision Low-vision Blind

69 Multistroke clusters 24 cap. Blind 138 iter. 22 cap. 126 iter Low-vision 11 cap. 65 iter. 10 cap. 60 iter

70 Study Limitations Difficulties in recruiting blind participants Under-representation of the younger generation Mean age of 48 (s.d.=15.8) No prior training sessions Some participants had limited used of touchscreen devices

71 Take-away recommendations

72 Recruit participants with the target disability

73 Longer gestures are curvier and perceived as more difficult

74 Avoid letter like figures WXYZ

75 Multi-finger gestures are more likely to go offscreen

76 Try to avoid time-based recognition

77 Use web technologies for multiplatform compatibility participants researchers

78 Thank you. Any questions?

ios Accessibility Features

ios Accessibility Features 1. Introduction Apple, since the birth of ios in 2007, has quietly been adding more and more sophisticated accessibility features to its mobile operating system. All of the following features are built