Designing Interactive Systems II

Transcription

1 Designing Interactive Systems II Computer Science Graduate Program SS 2011 Prof. Dr. Jan Borchers Media Computing Group RWTH Aachen University Jan Borchers 1

2 more abstract, user-oriented Review Applications User Interface Toolkit Window Manager Base Window System Graphics & Event Library Hardware Jan Borchers 2

3 Review User Interface Toolkit Window Manager Graphics and Event Library Hides hardware and OS aspects Drawing operations Event handling Base Window System Graphics & Event Library Jan Borchers 3

4 Review Base Window System Map n applications with virtual resources to 1 hardware Offer shared resources, synchronize access Windows & canvas, graphics contexts, color tables, events Event multiplexing and demultiplexing Window hierarchies Applications User Interface Toolkit Window Manager Base Window System Graphics & Event Library Hardware Jan Borchers 4

5 Today Window Manager User Interface Toolkit Jan Borchers 5

6 Window Manager: Motivation Position and decorate windows Provide Look&Feel for interaction with window system So far: applications can output to windows User control defined by application May result in inhomogeneous user experience Now: let user control windows Independent of applications User-centered system view BWS provides mechanism vs. Applications User Interface Toolkit Window Manager Base Window System Graphics & Event Library Hardware WM implements policy Jan Borchers 6

7 Window Manager: Structure Applications Application-independent user interface User Interface Toolkit Appearance ( Look ) Behavior ( Feel ) Look & Feel Window Manager Base Window System Tiling, Overlapping,... Pop-up menu at click Techniques Graphics & Event Library Hardware Request position change,... Fetch events Communicate with BWS Jan Borchers 7

8 Screen Management What is rendered where on screen? (layout question) Where empty space? What apps iconified? (practical q's) Example: Negotiating window position - Application requests window at (x,y) on screen; ignores position afterwards by using window coordinate system - BWS needs to know window position at any time to handle coordinate transformation, event routing, etc. (manages w) - User wishes to move window to different position - Or: Requested position is taken by another window Three competing instances (same for color tables,...) Solution: Priorities, for example: - Prior (app) < Prior (WM) < Prior (user) - WM as advising instance, user has last word Jan Borchers 8

9 Session Management Certain tasks are needed for all apps in consistent way Move window, start app, iconify window Techniques WM uses for these tasks Menu techniques - Fixed bar+pull-down (Mac), pop-up+cascades (Motif),... Window borders - Created by WM, visible/hidden menus, buttons to iconify/maximize, title bar Jan Borchers 9

10 Session Management WM techniques continued Direct manipulation - Manipulate onscreen object with real time feedback - Drag & drop,... - Early systems included file (desktop) manager in window manager; today separate standard application (Finder,...) Icon technique: (de)iconifying app windows Layout policy: tiling, overlapping - Studies showed tiling WM policy leads to more time users spend rearranging windows Jan Borchers 10

11 Window Manager Examples KDE 4.6 Gnome 3 Mac OS X 10.6 Jan Borchers 11

12 Session Management WM techniques continued Input focus: Various modes possible - Implicit (focus follows pointer): mouse/keyboard/... input goes to window under specific cursor (usually mouse) - Explicit (click to type): clicking into window activates it (predominant mode today) Virtual screens - Space for windows larger than visible screen - Mapping of screen into space discrete or continuous Jan Borchers 12

13 Virtual Desktops KDE Mac OS X Jan Borchers 13

14 Session Management WM techniques continued Look & Feel evolves hand-in-hand with technology - Audio, video I/O - Gesture recognition D windows (implemented by WM, BWS doesn't know) - Transparency To consider: - Performance hit? - Just beautified, or functionally improved? Jan Borchers 14

15 Late Refinement WM accompanies session, allows user to change window positions, etc. (changing app appearance) For this, application must support late refinement App developer provides defaults that can be changed by user Attributes must be publicized as configurable, with possible values App can configure itself using startup files (may be inconsistent), or WM can provide those values when starting app With several competing instances: priorities (static/dynamic!...) Jan Borchers 15

16 Levels of Late Refinement Per session, for all users System-wide information (table, config file,...) read by WM Per application, for all users Description for each application, in system-wide area Per application, per user Description file for each application, stored in home directory Per application, per launch Using startup parameters (options) or by specifying specific other description file Jan Borchers 16

17 Implementing Late Refinement Table files Key-value pairs, with priority rule for competing entries Usually clear text (good idea), user versions usually editable Modern versions: XML-based WM-internal database Access only via special editor programs Allows for syntax check before accepting changes, but less transparent; needs updating when users are deleted,... - Random Rant: Why Non-Clear-Text Config Files Are Evil Delta technique Starting state + incremental changes; undo possible Jan Borchers 17

18 Example: plist for login window application (Mac OS X) Jan Borchers 18

19 Window Manager: Location WM=client of BWS, using its access functions WM=server of apps, can change their appearance Several possible architectures WM as upper part of BWS - Saves comms overhead - But overview suffers WM as separate server - More comms - But exchangeable WM Jan Borchers 19 Apps BWS GEL HW WM Apps BWS GEL HW WM

20 Window Manager: Location Separate user process Uses mechanism of shared resources E.g., requests window position from BWS, checks its conformance with its layout policy, and requests position change if necessary Apps More comms, but same protocol as between apps & BWS no direct connection app WM BWS GEL HW WM Jan Borchers 20

21 Window Manager: Conventions Visual consistency For coding graphical information across apps Reduce learning effort Behavioral consistency Central actions tied to the same mouse/keyboard actions (right-click for context menu, -Q to quit) - predictability Description consistency Syntax & semantics of configuration files / databases consistent across all levels of late refinement Usually requires defining special language Jan Borchers 21

22 Window Manager: Conclusions WM leads from system- to user-centered view of WS Accompanies user during session Potentially exchangeable Allows for implementation of new variants of desktop metaphor without having to change entire system E.g., still much room for user modeling (see, e.g., IUI 2002) WM requires UI Toolkit to implement same Look&Feel across applications Jan Borchers 22

23 User Interface Toolkit Applications User Interface Toolkit Window Manager Base Window System Graphics & Event Library UIDS/UIDL Interface Guidelines (Look&Feel) Complex widgets Elementary widgets Hardware Motivation: Deliver API problem/user-oriented instead of hardware/bws-specific 50 70% of SW development go into UI - UITK should increase productivity Jan Borchers 23

24 UITK: Concept Two parts Widget set (closely connected to WS) UIDS (User Interface Design System to support UI design task) Assumptions UIs decomposable into sequence of dialogs (time) using widgets arranged on screen (space) All widgets are suitable for on-screen display (no post-desktop user interfaces) Note: decomposition not unique Jan Borchers 24

25 UITK: Structure Constraints User works on several tasks in parallel parallel apps Widgets need to be composable, and communicate with other widgets Apps using widget set (or defining new widgets) should be reusable Structure of procedural/functional UITKs Matched procedural languages and FSM-based, linear description of app behavior But: Apps not very reusable Jan Borchers 25

26 UITK: Structure OO Toolkits Widget handles certain UI actions in its methods, without involving app Only user input not defined for widget is passed on to app asynchronously (as seen from the app developer) - Matches parallel view of external control, objects have their own life Advantage: Subclass new widgets from existing ones Disadvantage: - Requires OO language (or difficult bridging, see Motif) - Debugging apps difficult Jan Borchers 26

27 UITK: Control Flow Procedural model: App needs to call UITK routines with parameters Control then remains in UITK until it returns it to app OO model: App instantiates widgets UITK then takes over, passing events to widgets in its own event loop App-specific functionality executed asynchronously in callbacks (registered with widgets upon instantiation) Control flow also needed between widgets Jan Borchers 27

28 Defining Widgets Widget : (W =(w 1...w k ),G=(g 1...g l ),A=(a 1...a m ),i=(i 1...i n )) Output side: windows W, graphical attributes G Input side: actions A that react to user inputs I Mapping inputs to actions is part of the specification, can change even at runtime Actions can be defined by widget or in callback Each widget type satisfied a certain UI need Input number, select item from list,... Jan Borchers 28

29 Simple Widgets Elementary widgets Universal, app-independent, for basic UI needs E.g., button (trigger action by clicking), label (display text), menu (select 1 of n commands), scrollbar (continuous display and change of value), radio button (select 1 of n attributes) Jan Borchers 29

30 In-Class Exercise: Button What are the typical components (W, G, A, I) of a button? Sample solution: W=(text window, shadow window) G=(size, color, font, shadow,...) A=(enter callback, leave callback, clicked callback) I=(triggered with mouse, triggered with key, enter, leave) Jan Borchers 30

31 Simple Widgets Container widgets Layout and coordinate other widgets Specification includes list C of child widgets they manage Several types depending on layout strategy Elementary & Container widgets are enough to create applications and ensure Look&Feel on a fundamental level Jan Borchers 31

32 Complex Widgets Applications will only use subset of simple widgets But also have recurring need for certain widget combinations depending on app class (text editing, CAD,...) Examples: file browser, text editing window Two ways to create complex widgets Composition (combining simple widgets) Refinement (subclassing and extending simple widgets) Analogy in IC design: component groups vs. specialized ICs Jan Borchers 32

33 Widget Composition Creating dynamic widget hierarchy by hierarchically organizing widgets into the UI of an application Some will not be visible in the UI Starting at root of dynamic widget tree, add container and other widgets to build entire tree Active widgets usually leaves Dynamic because it is created at runtime Can even change at runtime through user action (menus,...) Jan Borchers 33

34 Widgets and Windows The dynamic widget tree usually matches geographical contains relation of associated BWS windows But: Each widget usually consists of several BWS windows Each widget corresponds to a subtree of the BWS window tree! Actions A of a widget apply to is entire geometric range except where covered by child widgets Graphical characteristics G of a widget are handled using priorities between it, its children, siblings, and parent Jan Borchers 34

35 Refinement of Widgets Create new widget type by refining existing type Refined widget has mostly the same API as base widget, but additional or changed features, and fulfills Style Guide Not offered by all toolkits, but most OO ones Refinement creates the Static Hierarchy of widget subclasses Example: Refining text widget to support styled text (changes mostly G), or hypertext (also affects I & A) Jan Borchers 35

36 Late Refinement of Widgets App developer can compose widgets Widget developer can refine widgets User needs way to change widgets Should be implemented inside toolkit Solution: Late Refinement (see WM for discussion) Late refinement cannot add or change type of widget characteristics or the dynamic hierarchy But can change values of widget characteristics Jan Borchers 36

37 Style Guidelines How support consistent Look&Feel? Document guidelines, rely on developer discipline - E.g., Macintosh Human Interface Guidelines (but included commercial pressure from Apple & later user community) Limiting refinement and composition possible - Containers control all aspects of Look&Feel - Sacrifices flexibility UIDS - Tools to specify the dialog explicitly with computer support Jan Borchers 37

38 Types of UIDS Language-oriented Special language (UIL) specifies composition of widgets Compiler/interpreter implements style guidelines by checking constructs Interactive Complex drawing programs to define look of UI Specifying UI feel much more difficult graphically - Usually via lines/graphs connecting user input (I) to actions (A), as far as allowed by style guide Automatic Create UI automatically from spec of app logic (research) Jan Borchers 38

39 Language-Oriented UIDS <?xml version="1.0"?> <?xml-stylesheet href="chrome://global/skin/" type="text/css"?> <window id="findfile-window" title="find Files" orient="horizontal" xmlns=" <button label="normal"/>!! <button label="disabled" disabled="true"/> </window> Jan Borchers 39

40 Interactive UIDS Jan Borchers 40

41 more abstract, user-oriented Summary Applications User Interface Toolkit Window Manager Base Window System Graphics & Event Library Hardware Jan Borchers 41

42 A:$%52#02:;B!0"CC", >>FH=%-+0%.$%&".CCBI:1":$% 5*+.,J"7120#11"%>%&#:$%<<= ),1;0"*+;#02,"0K%D#L.#,%M"*+ C#L.#,F;"*+N0O2+B##*+",F?"!"#$%&'!(%)#*+",%-,./"01.23%4%567#0%8"*#2+76,%9:06;"%<=><%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%OOOF167#0F#0*+F0O2+B##*+",F?"%%

43 !"#$%&"'$(&)$*+%,$-"./&0$/"1$#2./$&)3456)&2./$37)$ (2&$ %&250"/'&$ "'$ 845")$!&."1/540$ 9:;9$ 20$ <"()2($ (2&#&'$ 201&)0"1240"5&0$ 81A(&01&0>&11=&>&)=$ 1)&1&0$ (2&$ "A#6&>B/51&0$ C02D&)#21B1&0$ '21$ 2/)&0$ ")./21&E1402#./&0$ F01>7)3&0$ 20$ ;:$ D&)#./2&(&0&0$!2#G2H520&0$6&6&0&20"0(&)$"0? -A36"=&$ (&#$ +&11=&>&)=#$ 2#1$ &#$ &20$ 0".//"5126&#$ +4/0E40G&H1$ GA$ &01>2.E&50I$ >&5./&#$ #2./$ (A)./$ 845")&0&)62&$ "A#G&2./0&1$ A0($ ("(A)./$ (&0$ 81"0(")($ &20&#$ J5A#KF0&)62&/"A#&#$ &))&2./1?$ L5&2./G&2126$ (2&01$ (&)$+&11=&>&)=$("GA$(2&$M33&0152./E&21$37)$("#$>2./126&$ C'6"06$'21$&)0&A&)=")&0$F0&)62&0$GA$#&0#2=252#2&)&0? C0#&)$ F01>A)3I$ ("#$ NO4A01&)$ F01)4HP$,4A#&QI$ ="#2&)1$ "A3$ (&)$ R(&&$ &20&#$ )&##4A).&0#./40&0(&0I$ &0&)62&4H12'2&)1&0$ S&=&0#GPE5A#I$ =&2$ (&'$ #4>4/5$ (2&$,&)#1&55A06$ "5#$ "A./$ 2/)$ %)"0#H4)1$ A0($ (2&$ #HB1&)&$ F01#4)6A06#'T652./E&21$ 6"0G/&2152./$ =&1)"./1&1$>&)(&0?$ U&=&0$ (&'$ A'3"06)&2./&0$ F20#"1G$ $ D40$ *&.P.5206H)4(AE1&0$ 'T./1&0$ >2)$ (&0$ N*&KC#&QK L&("0E&0$ "0$ 'T652./#1$ D2&5&0$ 81&55&0$ A0#&)&#$ 02./1$0A)$&20$(&A152./$)&(AG2&)1&)$J)2'B)&0&)62&=&(")3I$ #40(&)0$ "A./$ &20&$ #H"00&0(&I$ 20(2D2(A&55&$ ST#A06$ GAE703126&0$+4/0&0#?,"=&0$>2)$R/)$R01&)&##&$6&>&.E1$A0($>455&0$"A./$82&$ "'$ NO4A01&)$ F01)4HP$,4A#&Q$ 1&25/"=&0V$ +2)$ #1&/&0$ R/0&0$6&)0&$37)$>&21&)&$-A#E7031&$GA)$W&)376A06$A0($ 3)&A&0$A0#$"A3$&20&$&)3456)&2./&$XA#"''&0")=&21?