UI Tools, Techniques and Technologies

Transcription

1 UI Tools, Techniques and Technologies CS4HC3 / SE4HC3/ SE6DO3 Fall 2011 Instructor: Kevin Browne brownek@mcmaster.ca

2 UI Tools Some tools for building user interfaces... Content Management Systems (CMS) In-code library / API calls GUI builders User interface markup languages Constraint-based tools User interface modeling languages

3 UI Tools Distinctions between the different tools not black and white e.g. GUI builders may just graphically represent a user interface markup language Tool choice is not an issue of better or worse - different tools for different tasks A UI specified and generated by a CMS may be sufficient for an ecommerce website, a blog But the UI for a system of public transportation terminals may require advanced tools

4 UI Building Tool Concerns Level of abstraction... Targeting multiple platforms with same specification of the UI? Safety e.g. Java, modeling\constraint languages, CMS Typically comes at a cost.. Run-time slowdown Range of what can be specified Modelling language may allow one to detect bugs Economics How much time\money\resources do you have?

5 UI Building Tool Concerns Handling I/O How does the specification of the interface handle I/O? How much does the user control I/O? I/O handling largely automated? Event-driven functions are called when I/O events occur? Or must I/O be recognized and handled manually in the code by a loop searching for input? Economics How much time\money\resources do you have? Learning time? Extensibility? Modularity?

6 Content Management System Systems that manage workflow content... Repository of documents, pages Revising, publishing, storing documentation Supports collaboration User accounts Roles Typically handles security, backing up data, other housekeeping duties Make managing a system of documents easier

7 Content Management System UI separated from content database Front-end UI of CMS specified using the backend of the CMS Themes are typical... Can be customized Communities of users which develop themes CMS takes theme + settings + data and generates the UI automatically

8 Content Management Systems Great for non-technical or non-expert users Input/output handled automatically by CMS Explosive popularity on the web Economics likely a driving factor Database of managed content allows targeting of multiple platforms e.g. CMS built for web can target mobile devices Hamilton-area startup Weever Apps has built such a solution e.g. CMS Wordpress, Joomla!

9 GUI builder WYSIWYG editor for GUI creation Allow developer to drag-and-drop standard UI elements or widgets into the UI Widget parameters can be specified Actions and I/O involving widgets can be tied to event-handling functions Event-driven programming typical with GUI builders Often GUI builder is a nice front-end to a markup language or other interface representation

10 In-code library / API calls Creating the UI programmatically at run-time... Run-time environment such as OS creates standard GUI widgets given appropriate function calls e.g. Win32 API Graphics generated or displayed from standard library calls e.g. OpenGL Very often required to do non-standard GUI widgets in dynamic, graphically intensive software (i.e. Games, simulations)

11 In-code library / API calls Handling I/O When creating GUI widgets, may specify functions to handle widget events e.g. Interface actions mapped to function calls In case of standard graphics library, may have to handle I/O manually e.g. Game Engine style loop Debate when to use API calls vs GUI builder? If you require thousands of different dynamically customized pages... GUI builder tool likely to be impractical

12 User Interface Markup Language Markup language that specifies and renders GUI Most are XML dialects Rendered into GUI by an engine e.g. Interpreted into a document\webpage Meant to prevent re-invention of the wheel UI can be scripted Separates UI from function

13 User Interface Markup Language Examples: XML User Interface Language XUL Portable definition of common widgets for Mozilla applications User Interface Markup Language - UIML Doesn't describe how UI will look Instead specify UI widgets, how they should behave Attempts to be multi-platform, multi-language solution PDAs to websites

14 Constraint-based Tools Expressing a UI through constraints Constraint is a relationship that must be maintained Don't specify sequence of execution steps, but instead the properties of a UI solution to be found (i.e. what, but not how ) Static and temporal constraints Constraints can be maintained during execution and checked for error conditions beforehand e.g. Constraint-based Object-oriented Relations and Language (CORAL) Carnegie-Mellon

15 Constraint-based Tools Constraints can be used to... (Rozna, 1994) maintain consistency between underlying data and a graphical depiction of data on the screen to maintain consistency among multiple views on data to specify how information is to be formatted on the screen to specify animation events that are to occur when a given event occurs in the underlying system to specify attributes of objects in an animation, such as speed and trajectory

16 User Interface Modelling Language Need: good, standard notations for discussing design options English specification? Lengthy Vague Ambiguous Expression may be convenient for... Prove properties of the UI Completeness Correctness Consistency Checking for UI errors

17 User Interface Modelling Language Representation of UI, user actions with the software, how the software reacts User Interface Models: Capture requirements Avoid early commitment to particular designs Make roles, parts, relationships explicit Models are about specification, but techniques exist to automatically translate model into a UI UML? Can be done, but not really the intention of UML...

18 UMLi Extension of UML with better support for UI MB-UIDEs Model based user interface development environments Can design interfaces, but not applications UMLi combines UML + MB-UIDE Example:

19 Backus Normal Form (BNF) Non-terminals: high level components Terminals: specific strings <symbol> ::= expression expression... <symbol> is a non-terminal expression is a sequence of one or more symbols Sequences separated by, choice of possible substitution Symbols that never appear on left are terminals

20 Backus Normal Form (BNF) e.g. can specify user input... <character> = A B C D E... <string> = <character> <character><string> <lastname> = <string> <firstname> = <string>

21 Backus Normal Form (BNF) e.g. Can specify text oriented command sequences with multiparty grammars (Schneiderman, 2005) <session> ::= <U: Opening> <C:Responding> <U: Opening> ::= LOGIN <U: Name> <U: Name> ::= <U: string> <C: Responding> ::= Welcome [<U: Name>] Non-terminals which indicate the party that produces the string, square brackets indicate output In this case, U-user, C-computer

22 UI Prototyping Tools Paper mock-ups... Graphics editing programs... Photoshop, etc. Purpose built UI prototyping tools Often build what people will call the horizontal prototype of the system Confirm UI requirements, demo system to customers\investors Built to be quick, skinnable to different systems e.g. MockUp Screens (mockupscreens.com)

23 Game Loop Central to every video game Reasoning applicable to scientific simulations and other UI scenarios Video game UI executes and changes continuously and dynamically, as opposed to say a blog or Microsoft Excel while (not game over) Check for user input Update game state Display game

24 Game Loop Update game state involves: Artificial intelligence Movement of player, enemy units Resolving collisions Etc. Many things can be done in separate threads: User input may be done by separate thread, eventdriven style Sounds may be handled in different threads

25 Game Loop Implementation issue: Console titles (Xbox, etc.), earlier ios\smartphone software can assume virtually all computational power and resources are available Not so much for PCs, new multicore\multithread smartphone and tablets Frame Per Second number of times that the game loop executes (and displays the game) per second Problem: loop doesn't give indication of how we handle time

26 Game Loop One timing solution: fixed frames per second define FPS 30 while (not game over) starttime = getcurrentsystemtime() check for input update game state (time change (s) = 1 / FPS) display game starttime = starttime + (1 / FPS) sleep for ( max(0, starttime - getcurrentsystemtime() )

27 Game Loop Constant FPS solution Simple implementation! Always know exactly how much time should have changed since last update Problem... slow hardware Timing becomes variable Problem... fast hardware Could be executing at a higher FPS! If battery life is a concern, limiting CPU use to that required could be important i.e. Mobile, tablet devices

28 Game Loop Another timing solution: let game execute as fast as possible, update with time difference from previous loop execution while (not game over) check for input prevtime = currenttime currenttime = getcurrentsystemtime() update game state (currenttime - prevtime) display game

29 Game Loop Now the update function must account for variable lengths of times More difficult to implement, but not impossible... Problem: collision detection When any time difference is theoretically possible, how to detect collisions? Result: Objects going through walls, etc. Problem: using more resources than what is required for the game

30 Game Loop Other solutions: Set a minimum and maximum range for difference of time Only have to handle a specified range Makes handling collision detection easier Separate updating of game state from updating of display

31 Collision Detection Problem: detecting the collision of two or more objects Typically found in video games, scientific simulations of physics problems Range of design choices, sophistication levels Discrete vs. Continuous? Contact manifold? (set of intersecting points) Do objects have mass, acceleration? Other forces involved in the collision? (i.e. gravity) Computational geometry, linear algebra

32 Collision Detection Scientific simulations... Must simulate real-world or some specified system as precisely as possible Not necessarily in real-time... Low error rates are of paramount importance Video games... Must simulate physics in some reasonably acceptable way, in real-time Subjective player satisfaction is of paramount importance

33 Collision Detection Discrete collision detection Advance simulation by small steps Each time check for intersecting objects Or are close enough that we can reasonably say they are intersecting If objects are intersecting, then we account for the collision by modifying the position and trajectories of the objects a posteriori - we identify the collision after it occur

34 Collision Detection Continuous Compute trajectories of the objects Derive from this the instants that objects will collide Account for collision by modifying the position and trajectories of the objects a priori - we identify collision before it occurs

35 Collision Detection Discrete Algorithm is simpler, doesn't need to deal with time But intersections detected after collision has already occurred, must be fixed to improve accuracy Collisions can go undetected, objects can pass through one another, if they are moving fast enough! Checking for collisions must be relative to object speed Continuous Increased fidelity, stability More difficult to implement

36 Event-driven Programming Programming paradigm Widely adopted paradigm Control flow is determined by events events could be actions performed by the user, or messages from other threads Main Loop (aka Event Loop) Event selection Waits for event messages (typically blocks) to arrive, selects an event to handle Event handling Typically operates asynchronously in its own thread

37 Event-driven Programming Event handler Functions or subroutines Handle events after called by main loop Somehow (usually automatically by the IDE) are bound to an event Functions themselves are usually created automatically by IDE, developer just has to insert code Sometimes an event object is provided where the developer can access properties of the event e.g. The event might be that a key was pressed, event.keypressed() will return the key character

38 References Designing the User Interface: Strategies for Effective Human-Computer Interaction / 5 th edition, by Ben Schneiderman & Catherine Plaisant (2010) Chapman, Roger. An Instructor's Outline of Designing the User Interface 4 th Edition (2004) by Ben Schneiderman & Catherine Plaisant. (slides) Paperin, Greg. Intersection Testing in 3D Graphics. (2006) University of Hamburg, Germany. paperin.org P. Pinheiro da Silva, N. W. Paton. User Interface Modeling in UMLi. (2003) Stanford University / University of Manchester. Rozsa, Eniko Ilona. Design and constraint visualization in computer-aided design. (1994) Simon Frasier University. M.A.Sc. Thesis Witter, Koen. DeWiTTERS Game Loop.(2009)