UI Toolkits. HCID 520 User Interface Software & Technology

Size: px

Start display at page:

Download "UI Toolkits. HCID 520 User Interface Software & Technology"

Stanley Webb
6 years ago
Views:

1 UI Toolkits HCID 520 User Interface Software & Technology

3 Xerox Alto 1973

4 Evolution of User Interfaces Command Line (UNIX shell, DOS prompt) Interaction driven by system User prompted for input when needed Text-based input and output Event-Driven Interfaces (GUIs) Interaction driven by user System waits for user action and responds Pointing & text input, graphical output More complex architecture & development

5 Component Model

6 Component Model Encapsulate interactive components (widgets) Component library (button, slider, container) Interface built as a hierarchy of components Widgets drawn by underlying graphics library Input event generation and dispatch Historically mouse & keyboard, now touch, Bounds management & damage/redraw Model geometry, redraw updated regions only

8 Java Swing Widgets

10 User Interface Components Label TextArea Each component is a clipped 2D Buttons canvas with its own coordinate system.

11 User Interface Components Each component (widget) is an object with: A bounding box A paint method for drawing itself Drawn in the component s coordinate system Callbacks to process input events public void paint(graphics g) { g.fillrect( ); // interior g.drawstring( ); // label g.drawrect( ); // outline }

12 Containment Hierarchy Window Panel Label TextArea Panel Button Button

13 Component Layout Each container is responsible for positioning its contents. Label Window Panel TextArea Border Layout (direct placement) Panel NORTH Button Button springs CENTER struts Struts and Springs (simple constraint-based layout) SOUTH

14 Event-Driven Architecture

15 Events User input is modeled as events that must be processed by the system. Hardware Events Mouse: move, button-down, button-up Keyboard: key-down, key-up Inferred Events Mouse: click, double-click, enter, exit, drag Keyboard: key-press Window: move, resize

16 Anatomy of an Event Events encapsulate the information needed for components to react to input or changes. Event Type (mouse moved, key down, etc) Event Target (the input component) Timestamp (when did event occur) Modifiers (Ctrl, Shift, Alt, etc) Event Content: Mouse: x,y coordinates, button pressed, # clicks Keyboard: which key was pressed

Find proper component(s) Invoke callbacks on components Repeat, or wait until event

17 Event Dispatch Loop Event Queue List of input events Mouse moved (t 0,x,y) Event Loop (runs in dedicated thread) Remove next event from queue Determine event type Find proper component(s) Invoke callbacks on components Repeat, or wait until event arrives Component Invoked callback method Update application state Request repaint, if needed

18 Event Dispatch Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Window Panel Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) (queue and dispatch incoming events in a dedicated thread) Label TextArea Panel /* callback for TextArea */ public void mousemoved(e) { // process mouse moved event } Button Button

19 Demo: Browser DOM Events Go to Open the JavaScript console, enter: var h = document.queryselector("#header"); h.addeventlistener("click", function(event) { console.log(event); }); Click the header image at the top of the page. Inspect the events in the JavaScript console.

21 Event Minutiae Event Capture Process event either as it moves down or up the component hierarchy? Default is up. Why?

22 Event Capture (Down) Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Window Panel Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) (queue and dispatch incoming events in a dedicated thread) Label TextArea Panel /* callback for TextArea */ public void mousemoved(e) { // process mouse moved event } Button Button

23 Event Bubbling (Up) Window Panel Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) (queue and dispatch incoming events in a dedicated thread) Label TextArea Panel Button Button

24 Event Minutiae Event Capture Process event either as it moves down or up the component hierarchy? Default is up. Why? Throttling & Debouncing You don t always want to process every event, both for performance and experience reasons. Examples: Flurry of mouse drags > process last event Rapid enter/exit > pause to process exit

25 Model View Controller

26 Model View Controller (MVC) Architecture for interactive applications Introduced by Smalltalk developers at PARC Partitions app to be scalable & maintainable Model View Controller

27 Model Information the app is trying to manipulate Circuits: logic gates and connecting wires Drawing: geometry and color of shapes Model View Controller

28 View Implements a visual display of the model Apps may have multiple views for one model Example: shape & numerical views of drawing Model View Controller

29 View Implements a visual display of the model When model changes, each view must be notified so that it can update itself. Model View Controller

30 Controller Processes all input events from the user Query view to interpret events (e.g., selection) Call methods of model to make changes Model updates & notifies views Model View Controller

31 Example Drawing Application Blue circles: 3 Red squares: 2

32 The Controller handles input Blue circles: 3 Red squares: 2

33 updates the Model Click Blue circles: 3 Red squares: 2

34 and Views update in response. Blue circles: 4 Red squares: 2

35 Relationship of View & Controller Controller must contact view to interpret what some user events mean. Example: Mouse(x, y) -> selected object Pattern of behavior in response to user events (controller issues) is independent of visual geometry (view issues).

36 Combining View & Controller View and controller are tightly intertwined Lots of communication between the two Occur in pairs: each view needs a controller Some architectures combine into a single class Model View Controller

37 Why MVC?

38 Why MVC? Putting MVC into one class does not scale Model may have more than one view Each view different, update on model change Separation aids maintenance & extensibility Easy to change view or add a new view later Models can be extended; old views still work Flexibility of input handling when using separate controllers (e.g., mouse vs. touch)

39 Adding Views Later Blue circles: 3 Red squares: 2

40 Changing the Display Erase and Redraw Using background color to erase fails Draw shape in new position -> loses ordering Move in model and then redraw everything Change position of shapes in model Model keeps shapes in a desired order Tell all views to redraw themselves in order Slow for large / complex drawings

41 Move a red square Blue circles: 3 Red squares: 2

42 Erase at old position, draw at new Blue circles: 3 Red squares: 2

43 Register damage, redraw region Blue circles: 3 Red squares: 2

44 Damage / Redraw View informs window system of areas that need to be updated (i.e., damaged) Does not redraw these areas yet Window system batches updates & clips them to visible portions of window On next event cycle, window system invokes repaint, passes region that need updates Critical for many 2D graphics systems, typically not used in 3D interfaces. Why?

45 Cascading Style Sheets

46 Technical Limitations Traditional GUI tools often make assumptions: Rectangular (non-overlapping?) components Difficult to style graphic elements Many modern frameworks are more flexible, using a scenegraph to model the display. Ex: the browser document object model (DOM). Decouple content from appearance using separate stylesheets to specify design.

47 Cascading Style Sheets A language for styling an HTML document W3C standard, first introduced in 1996 Goal: separate content from presentation Specify layout, spacing, colors, fonts, Share formatting across pages on a site Enable various formats for a single page Why cascading? Multiple style definitions may apply to an element, ordered ( cascaded ) by priority.

48 CSS Example p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333; }

49 CSS Example Selector style p (paragraph) elements p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333; }

50 CSS Example Selector p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333; } Property

51 CSS Example Selector p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333; } Property Value or #333, or rgb(51,51,51)

52 CSS Selectors Tag (select all elements of a certain type) a, p, h1, ul, li, etc

53 CSS Selectors Tag (select all elements of a certain type) a, p, h1, ul, li, etc Id (select by unique identifier, prefix #) <div id="logo"> > #logo

54 CSS Selectors Tag (select all elements of a certain type) a, p, h1, ul, li, etc Id (select by unique identifier, prefix #) <div id="logo"> > #logo Class (select by shared identifier, prefix.) <div class="c1 c2"> >.c1

55 CSS Selectors Tag (select all elements of a certain type) a, p, h1, ul, li, etc Id (select by unique identifier, prefix #) <div id="logo"> > #logo Class (select by shared identifier, prefix.) <div class="c1 c2"> >.c1 These selectors can be combined: div.headlines

56 CSS Selector Combinations For multiple selections, use a comma div.headlines, div.news { } Selects both div.headlines and div.news elements

57 CSS Selector Combinations For multiple selections, use a comma div.headlines, div.news { } Selects both div.headlines and div.news elements For nested selections, use a space div.headlines ul li { } Selects all li within a ul within a div.headlines

58 CSS Selector Combinations For multiple selections, use a comma div.headlines, div.news { } Selects both div.headlines and div.news elements For nested selections, use a space div.headlines ul li { } Selects all li within a ul within a div.headlines For direct descendants, use > div.headlines > ul li { } Now the ul must be the direct child of div.headlines

59 CSS Box Model

60 CSS display property block <div> inline <span> inline-block none

61 Specify a 2-Column Layout <body> <div id= left > Sidebar column content (assume max 400px tall) </div> <div id= right > Main column content (variable height) </div> <div id= footer > Footer column content (assume 50px tall) </div> </body>

62 Specify a 2-Column Layout <body> > select with body <div id= left > > select with #left Sidebar column content (assume max 400px tall) </div> <div id= right > > select with #right Main column content (variable height) </div> <div id= footer > > select with #footer Footer column content (assume 50px tall) </div> </body>

63 Layout with no styling

64 Layout with absolute positioning, 250 pixel sidebar What are some shortcomings of this layout? Why did we include body { position: relative }?

65 Layout with absolute positioning, without position:relative position: relative sets the coordinate space for absolutely positioned elements. In this example the position [0,0] is relative to the browser window, not the body tag.

66 Layout with float: left and margin-left What are some shortcomings of this layout? Why did we include #footer { clear: both }?

67 Layout with float: left and margin-left, without clear: both clear: both tells the browser layout engine to clear any floats (both left and right) before placing the element. Next, can we make the sidebar resize automatically in response to the window width?

68 Float layout using percentages What if we want to ensure the width of the left sidebar never falls below a minimum size?

69 Float layout using percentages and minimum width Here s one attempt at setting a minimum width. Why is this unsatisfactory?

70 Float layout using percentages Let s go back to our previous design How can we limit the maximum width of the page?

71 Float layout using percentages and body max-width Use the max-width property on the body element.

72 Adding automatic margins margin-left: auto; margin-right: auto centers the body.

73 Useful CSS Properties margin (-left, etc) padding (-left, etc) border (-left, etc) border-radius overflow (-x/-y) background-color background-image position float (max-/min-) width, height font font-family font-size font-weight font-variant line-height text-transform text-align vertical-align display

74 CSS Power Tools CSS preprocessors (LESS / SASS) Add support for variables (e.g., $basecolor) Perform computation, unit conversion Enable better reuse, less repetition Now commonly used in web applications HTML/CSS frameworks (e.g., Bootstrap) Provide better defaults out of the box More dynamic components (popup menus) Responsive design; mobile friendly

75 UI Tools Discussion

76 Myers, Hudson & Pausch Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. At any point did they miss the mark?

77 Myers, Hudson & Pausch Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. At any point did they miss the mark? Web: subsuming many desktop applications Design vs. Dev: In practice often done by teams with varied skills (interaction design, graphic design, implementation) 3D: Mostly limited to games? BUT fabrication now likely to drive new 3D design tools

78 Toolkits Abound Not just for desktop GUI widgets & events Mobile applications (ios, Android) Information visualization (d3.js, Protovis, Prefuse) Tangible user interfaces (Papier-Maché) Gestural user interfaces (Kinect API) Speech user interfaces (MS Speech API) Proxemic user interfaces (Proximity Toolkit) Many exhibit similar tensions around expressiveness, efficiency and ease-of-use

79 Toolkit Benefits & Shortcomings Simplify & accelerate UI development Lower the threshold of specification difficulty Raise the ceiling of what can be expressed But hard to do both Establish path of least resistance Makes common / successful designs easy Makes unusual / ineffective designs harder Can limit & shape what we conceive / design Enforce consistency of design Maintain familiar widgets and interactions

80 Discussion Question Topics Timeliness & Future Predictions Threshold/Ceiling, Expressivity vs. Usability Varied Display Sizes (from Walls to Watches) Personalization User Attention & Cognitive Load End User Programming Constraints / Model-Driven / Automatic Design Security

81 The authors of the main reading mention the possibility of tools that not only enable design systems that have high usability, but that actual promote or help ensure that they are. How might this be taken further in the future to help prevent designs that cause users frustration or discomfort (or error)? - Avery The author suggests that there should be a balance between creating UIs that are accessible while also allowing for innovation to accommodate for ubiquitous computing. Shouldn't UI design and technology revolve around how a task is performed as opposed to creating technologies to fit a certain form factor or input/output interaction? - Garrick

UI Toolkits. HCID 520 User Interface Software & Technology

UI Toolkits. HCID 520 User Interface Software & Technology UI Toolkits HCID 520 User Interface Software & Technology http://www.cryptonomicon.com/beginning.html Xerox Alto 1973 Evolution of User Interfaces Command Line (UNIX shell, DOS prompt) Interaction driven