CS260 UI Toolkits Björn Hartmann University of California, Berkeley EECS, Computer Science Division Fall 2010
In the beginning cryptonomicon.com/beginning.html
The Xerox Alto (1973)
Event-Driven UIs Old model (e.g., UNIX shell, DOS) Interaction controlled by system, user queried for input when needed by system Event-Driven Interfaces (e.g., GUIs) Interaction controlled by user System waits for user actions and then reacts More complicated programming and architecture
Console program pseudo-code Do some work Prompt user for input Wait for user input Process user input Do some more work Exit
Console program // Java Example: Console console = System.console(); String name = console.readline( Your name: ); System.out.println("You have entered : " + name);
Minimal interactive program Do until a quit command: { " wait for user input " process it " (optionally) update display }
Minimal interactive program Do until a quit command: { " wait for user input " switch (input-cmd) {!! case insert: do-insert( )!! case delete: do-delete( )!! case backspace: " (optionally) update display }
Minimal interactive program Can t use this (global) approach for window systems, because the result of a user command depends on the active window (and the active component within that window). Too many possible combinations of input x target window.
Widgets Encapsulation and organization of interactive controls Class hierarchy encapsulating widgets Top-level Component class Implements basic bounds management, and event processing Drawn using underlying 2D graphics library Input event processing and handling Typically mouse and keyboard events Bounds management (damage/redraw) Only redraw areas in need of updating
W. Newman Light Handle 1967/68 11
Interface Builder - Library
Toolkit Example: Java Swing GUI toolkit with a widget set and an API 7
Declarative Layout (WPF) <StackPanel> <Label>Enter Text:</Label> <TextBox TextWrapping="Wrap > </TextBox> <StackPanel Orientation="Horizontal" HorizontalAlignment="Right"> <Button>Ok</Button> <Button>Cancel</Button> </StackPanel> </StackPanel>
User Interface Builders 8
Event Dispatch Apple, Cocoa Event-Handling Guide
Event Dispatch Loop Event Queue Queue of input events Mouse moved (t 0,x,y) Event Loop (runs in dedicated thread) Remove next event from queue Determine event type Find proper component(s) Invoke callbacks on components Repeat, or wait until event arrives Component Invoked callback method Update application state Request repaint, if needed
Cocoa Event Dispatch Loop 1) Events from input devices enter here 3) Main loop processes one event per iteration 2) Event is added to FIFO event queue Apple, Cocoa Event-Handling Guide
Event Dispatch: Targeting Label TextArea Buttons
Event Dispatch Window Panel Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, F1 ) (queues and dispatches incoming events in a dedicated thread) Label TextArea Panel /* callback for TextArea */ public void mousemoved(e) { // process mouse moved event } Button Button
Mouse/Touch vs. Keyboard Events Mouse Events are (usually) routed to the top-most (in z- order) visible component underneath the cursor using hit testing. Exception: captured mouse events after beginning interaction Keyboard events are (usually) routed to the component that has key focus. Exceptions: keys that change focus, accelerator keys
Success of Tools Window Managers, User Interface Toolkits, and Interface Builders are ubiquitous Most software built using them Are based on many years of HCI research: Brad A. Myers. A Brief History of Human Computer Interaction Technology. ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. 11
Why Tools? The quality of the interfaces will be higher. Why? Rapid prototyping. Easier to incorporate changes motivated by evaluation. Re-use affords investment in high quality tools. Consistency of interface design. Enable collaboration among specialists. 12
Why Tools? The UI will be easier to create and maintain. Why? Less code to write due to component re-use. Better modularization, separation of concerns Tools may abstract complex systems or algorithms. Easier to port an application to different hardware or software environments if device dependencies are isolated in the user interface tool. 13
What should tools do? Help design the interface given a specification of the tasks. Help implement the interface given a design. Help evaluate the interface after it is designed. Create easy-to-use interfaces. Allow the designer to rapidly investigate different designs. Allow non-programmers to create user interfaces. Provide portability across different machines and devices. Be easy to use themselves. 14
Tools might do: Provide sets of standard UI components Guide the implementation Help with screen layout and graphic design. Validate user inputs Handle user errors Handle aborting and undoing of operations Provide help and prompts Deal with field scrolling and editing Insulate the application from all device dependencies Allow the end user to customize the interface. 15
Application Types What application domains are deserving of specialized toolkit support? 16
Application Types Word processors Drawing programs (CAD/CAM) Painting programs Mail readers Spreadsheets Programming environments / code editors WWW browsers Interactive games Visualizations Automated-teller machines (ATM) Virtual Reality Multi-media Video Animation Controlling machinery 17
DiamondSpin Toolkit Toolkit for tabletop UIs (Shen, Vernier, Forlines, Ringel, CHI 04) 18
19
Tabletop UI Needs Multi-user support Identity-aware widgets Multiple menus Public and personal spaces Resolving conflicting actions Arbitrary orientation of UI elements Techniques to control orientation and layout Rotation sensitive components 20
istuff Toolkit Physical UI components for ubiquitous computing environments (multiple users, devices, and applications) [Ballagas, Ringel, Stone, Borchers], CHI 03 21
istuff Design istuff components Device + proxy ( smarts are in the proxy) PatchPanel Translate between istuff events and applicationspecific events Run-time retargetable events Address dimension mismatches 22
23
Discussion of Themes Address the useful & important aspects of UIs Narrower tools have been more successful than ones that try to do everything Do one thing well Threshold / Ceiling Research systems often aim for high ceiling Successful systems often seem to instead aim for a low threshold Impossible to have both? 24
Threshold & Ceiling Programming in C Visual Basic Director (v6) Difficulty of Use MFC Click and Create C Programming Lingo xcmds HyperCard C Programming HyperTalk Basic Goal Sophistication of what can be created 25
Discussion of Themes, cont. Path of Least Resistance Tools should guide implementers into better user interfaces Goal for the future: do this more? Predictability Programmers do not seem willing to release control Especially when system may do sub-optimal things Moving Targets Long stability of Macintosh Desktop paradigm has enabled maturing of tools 26
Evaluating User Interface Tools An API is a user interface where programmers are the users Should you evaluate toolkit as you would a UI? Factors Expressiveness (Ceiling) Development Rate (of skilled user) Learning Rate (to become skilled) Performance Portability 27
Class Deployment (In collaboration with Leapfrog)
41
42
43
44
The Future of Interface Tools Supporting Prototyping Collaboration Evaluation of interfaces built with tools of tools themselves how to prototype, test, iterate? Emerging interface styles, such as mobile recognition-based UIs (speech, pens, vision) multiple devices 28
Summary Toolkits provide reusable interface components to simplify UI development Toolkit trap: it s tempting to only make UIs that the toolkit makes easy, instead of making what s best for a specific app Toolkit types: WIMP (Garnet, Swing, Motif, etc) Specialty (Phidgets, istuff, Papier-Mache, DiamondSpin, GroupKit, Peripheral Displays Toolkit, etc) 30
Toolkit features may not matter... Arduino: no smart components programmed in C Wildly successful with amateurs & artists. Why? 48