UI Toolkits. HCID 520 User Interface Software & Technology

Transcription

1 UI Toolkits HCID 520 User Interface Software & Technology

2

3 Xerox Alto 1973

4 Evolution of User Interfaces Command Line (UNIX shell, DOS prompt) Interaction driven by system User prompted for input when needed Text-based input and output Event-Driven Interfaces (GUIs) Interaction driven by user System waits for user action and responds Pointing & text input, graphical output More complex architecture & development

5 Component Model

6

7

8

9 Component (Widget) Model Encapsulate interactive components (widgets) Component library (button, slider, container) Interface built as a hierarchy of components Widgets drawn by underlying graphics library Input event generation and dispatch Historically mouse & keyboard, now touch, Bounds management & damage/redraw Model geometry, redraw updated regions only

10 User Interface Components Each component (widget) is an object with: A bounding box A method for drawing itself Drawn in the component s coordinate system Methods (callbacks) to process input events public void paint(graphics g) { g.fillrect( ); // interior g.drawstring( ); // label g.drawrect( ); // outline }

11 User Interface Components Label TextArea Each component has its own 2D coordinate system. Buttons

12 Containment Hierarchy Window Panel Label TextArea Panel Button Button

13 Event-Driven Architecture

14 Events User input is modeled as events. Events are dispatched to components for processing.

15 Events User input is modeled as events. Events are dispatched to components for processing. Hardware Events Mouse: move, button-down, button-up Keyboard: key-down, key-up

16 Events User input is modeled as events. Events are dispatched to components for processing. Hardware Events Mouse: move, button-down, button-up Keyboard: key-down, key-up Inferred Events Mouse: click, double-click, enter, exit, drag Keyboard: key-press Window: move, resize

17 Anatomy of an Event Events encapsulate the information needed for components to react to input or changes. Event Type (mouse moved, key down, etc) Event Target (the input component) Timestamp (when did event occur) Modifiers (Ctrl, Shift, Alt, etc) Event Content: Mouse: x,y coordinates, button pressed, # clicks Keyboard: which key was pressed

18 Demo: Browser DOM Events Go to Open the JavaScript console, enter: var h = document.queryselector("header"); h.addeventlistener("click", function(event) { console.log(event); }); Click the header region at the top of the page. Inspect the events in the JavaScript console.

19

20 Event Dispatch Loop Event Queue List of input events Mouse moved (t 0,x,y) Event Loop (runs in dedicated thread) Remove next event from queue Determine event type Dispatch to proper component(s) Invoke callbacks on components Continue, or wait until event arrives Component Invoked callback method Update application state Request repaint, if needed

21 Event Dispatch Loop

22 Event Dispatch Loop Mouse moved (t 0,x,y)

23 Event Dispatch Loop Event Queue List of input events Mouse moved (t 0,x,y)

24 Event Dispatch Loop Event Queue List of input events Mouse moved (t 0,x,y) Event Loop (runs in dedicated thread) Remove next event from queue Determine event type Dispatch to proper component(s) Invoke callbacks on components Continue, or wait until event arrives

25 Event Dispatch Loop Event Queue List of input events Mouse moved (t 0,x,y) Event Loop (runs in dedicated thread) Remove next event from queue Determine event type Dispatch to proper component(s) Invoke callbacks on components Continue, or wait until event arrives Component Invoked callback method Update application state Request repaint, if needed

26 Event Dispatch Window Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) Panel Label TextArea Panel Button Button

27 Event Dispatch Window Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) Panel Label TextArea Panel Button Button

28 Event Dispatch Window Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) Panel Label TextArea Panel Button Button

29 Event Dispatch Window Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) Panel Label TextArea Panel Button Button

30 Event Dispatch Window Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) Panel Label TextArea Panel Button Button

31 Event Dispatch Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Window Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) Panel Label TextArea Panel /* callback for TextArea */ function mousemoved(evt) { // process mouse moved event } Button Button

32 UI Tools Discussion

33 Myers, Hudson & Pausch Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. At any point did they miss the mark?

34 Myers, Hudson & Pausch Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. At any point did they miss the mark? Web: subsuming many desktop applications Design vs. Dev: In practice often done by teams with varied skills (interaction design, graphic design, implementation) 3D: Mostly limited to games? BUT fabrication now likely to drive new 3D design tools. VR?

35 Toolkits Abound Not just for desktop GUI widgets & events Mobile applications (ios, Android) Information visualization (d3.js, Protovis, Prefuse) Tangible user interfaces (Papier-Maché) Gestural user interfaces (Kinect API) Speech user interfaces (MS Speech API) Proxemic user interfaces (Proximity Toolkit) Many exhibit similar tensions around expressiveness, efficiency and ease-of-use

36 Toolkit Benefits & Shortcomings Simplify & accelerate UI development Lower the threshold of specification difficulty Raise the ceiling of what can be expressed But hard to do both! Establish path of least resistance Makes common / successful designs easy Makes unusual / ineffective designs harder Can limit & shape what we conceive / design Enforce consistency of design Maintain familiar widgets and interactions

37 Discussion Question Topics Threshold/Ceiling, Expressivity vs. Usability Recognition-Based UIs Multi-Device Interaction 3D UIs / Augmented and Virtual Reality Timeliness & Future Predictions The Moving Target Problem Accessibility Role of Tools in Collaboration & Communication

38 Clarifications What are event languages and object oriented programming? What is the difference between scripting languages and compiled languages? Clarify: the trick that has been used by CS researchers of buying expensive, high performance computers to investigate what will be available to the public in five or 10 years no longer works

39 Questions How has responsive web design evolved to meet the needs of devices with varying input and output capabilities? - Christopher Chung Do you think that some techniques that did not catch on in the past - such as model-based and automatic techniques - could provide viable and effective UI tools in the current landscape of technology? Why or why not? - Fannie Jou I wonder, how much more or less effective would interfaces be if designers had stayed clear of skeuomorphism and allowed the digital world to develop as something far removed from the expectations of pen and paper interactions? - Megan Taylor

40 More Questions Were the authors wrong in their prediction that handwriting interpretation would play a pivotal role? Is speech a preferable form of natural language input that leapfrogged handwritten input? - Lyle Kyne Is it feasible to consider a system that requires no understanding of the underlying code and technology that allows the construction of an interface and interactions to suit a specific person, eg a nurse, mechanic, schoolteacher, etc? - Kyle Freed [A]re user interface software tools really as important as argued in the article? - Elsa Ho

41 EXERCISE 1 Widget Design

42 Hello

43 Default State Hello

44 Default State Data: styling, content, etc. Hello

45 Default State Data: styling, content, etc. Hello form-checkboxes-and-radio-buttonssky-ui-kit-psd/ FID=9&TID=10550 Origin Unknown.

46 Checked State Hello FID=9&TID=10550 Origin Unknown.

47 Disabled-Unchecked Disabled-Checked Focus Maybe-Checked HTML/Forms/The_native_form_widgets FID=9&TID=10550 Origin Unknown. library/ui-elements/html5- checkboxes

48

49 State Diagram to the Rescue Click Checked Disabled Default Click

50 State Diagram to the Rescue Click Checked Disabled Default Click Click Maybe

51 State Diagram to the Rescue Click Checked Disabled Default Click Shift- Click Click Maybe

52 Your Turn! Submit! In pairs, brainstorm the states + transitions for a submit button.! What does the button look like in each state?! What are the transitions between states?! What actions trigger the transitions?! How does a user know what to do?! Regroup in ~10 mins to share ideas.

53 Your Turn! I am a textbox! In pairs, brainstorm the states + transitions for a textbox.! Hint: States+transitions for the textbox but also its contents (e.g., text selection).! Regroup in ~10 mins to share ideas.

54 Your Turn: Interactive Prototypes