Workshop Some reminders about user experience and usability evaluation. TIE project course Thomas Olsson

Transcription

1 Workshop Some reminders about user experience and usability evaluation TIE project course Thomas Olsson

2 Aspects of the product to evaluate from user s perspective Functionality: Effectiveness, match with needs, possibilities, benefits, coverage, appropriateness, risks Interaction, input-output techniques: Suitability/preferences, user experience, learnability, intuitiveness Information content: Reliability & credibility, stimulation, aesthetics, usefulness, timeliness & being up-to-date UI: Usability, look & feel, navigation & workflow Overall quality : User acceptance, preferences, comparison with other similar systems, intention to use in future + Packeting, Customer service, Brand, Cost-Quality ratio etc.

3 What to measure? Usability (usually rather specifically operationalized) Efficiency (objective?) Task completion & quality of outcome How well did a task succeed, how high-quality is the work that was done with the system? Appropriateness / suitability (more subjective?) Ease-of-use: learnability, understandability, match with users conceptual models User experience, satisfaction (subjective) Positive outcomes, pleasurability, fun, mental stimulation, evocation of memories, social values, product attachment, inspiration & surprise, self-presentation + many more Effectiveness in creating emotions Technology acceptance: intention to accept and put into use perceived ease of use, perceived usefulness, relative advantage, compatibility, voluntariness of use, trust Aspects specific to the domain or application area: Playfulness, sociability, privacy, etc. Mobile-UI vs. stationary Specific user groups

4 Concepts and approaches related to UX testing Objective metrics: task completion %, # of errors, time consumption, # of clicks Often these are quantitative, i.e. numerical, aiming to generalizations Subjective metrics: opinions, subjective agreement statements, user experiences Often these are qualitative, i.e. descriptive, not as easily measurable & quantifiable Formative evaluation: guiding & supporting the development, evaluating details of design Summative evaluation: summarizing, measuring general goodness & experiences & implications ( potency, efficacy ) In lab vs. in situ : how to make the test situation as authentic as possible? During tasks vs. in retrospect (e.g. think aloud + final interview) Comparisons: e.g. expectations vs. experience

5 Some pointers regarding methods Some relatively simple-to-use measures: Usability SUS: QUIS: SUMI: Mental task load (NASA-TLX) Generic UX metrics: AttrakDiff: User experience survey (google it) User acceptance: E.g (see appendix) Subjective emotions: PrEmo, SAM, Geneva Emotion Wheel, PANAS, MIS

6 Some pointers regarding methods Approaches: Selecting the preferred option, ordering according to preference Scales of measuring in surveys Agreement scale (Likert): Totally agree (totally disagree) Semantic differentials: Innovative Old-fashioned Specific dimensions: (e.g. frequency) Always frequently every now and then seldom never Using baselines & benchmarks Attributing words (e.g. adjectives), sentence completion How would you describe the quality of this app Heuristics & walkthrough methods Observation while using + Interviewing Think aloud: momentary viewpoints, subjective, unfocused, requires clarification

7 Additional readings Kasper Hornbæk (2006) Current practice in measuring usability: Challenges to usability studies and research PG Zimmermann (2008) Beyond Usability Measuring Aspects of User Experience. Doctoral dissertation. sbagozzi.pdf

8 TIE Project Work on Pervasive Systems Workshop on UX evaluation, Thomas Olsson Groups answers to warm-up questions How will you validate that the product matches the needs and requirements of the target user and the context of use? With the help of or by the customer: - We trust our customer s requirements and experience of ux. We don t know yet application s end users. (G10) - We deliver the product to the "end user" (software customer) after every sprint and evaluate it together (which is usually more about architecture and non-graphical quality). There will probably be no users for our system. (G2) - we plan to go through the steps of using the product with the customer. The most important requirement is easy-of-use, so the usability issues and deviations should be discovered there. (G1) - We are doing our application agile way with really short iterations. Thanks to this we get feedback from our customer's contact persons (which includes some of the end users) quite fast and are able to change the features they don't like and add new things that they suggest. Also somewhere in the future we are most likely also going to test our application's usability with some of the end users who do not yet know how our application works. (G6) - There are no clear requirements for usability because of the nature of the application (mashup of 3 different applications), except that it should be easy enough for the end users to use. The usability will be verified by our client (G4) User involvement: - By defining the targeted user group (users groups) at the early phase of the development process, then we will conduct heuristic evaluation with expert users so we could have external point of view if the product matches the needs of the user's group (users' groups). By conducting some usability test with candidates selected in our targeted users' groups and observe how they perform the tasks (tasks that follow the requirements). By also collecting and analysing users' feedbacks trough web surveys. (G7) - By testing the product in the real environment with the end users (G8) - Actual users are going to test the product from early versions in actual environment. (G5) Very detailed plans, awesome! - Our product Kactus 3D is to create a new kind of 3D visualization for the architectural SoC and embedded system designs. After we build up the prototype of our product, a preliminary evaluation plan will be involved in the procedure of our project. For the purposes of evaluation of needs and requirements, we will seek help from 1-3 experienced

9 participants. All subjects are requested as a customer/professor/researcher in computer system in TUT. They will evaluate our prototype from a user perspective (professional FPGA/SoC designer) whether SoC and FPGA developers easily manage and explore complicated design hierarchies, whether using the implementation of a vivid visualization demo attests the feasibility and effectiveness of 3D architectural SoC designs. If the answers are yes, which means our prototype basically meets the needs and requirements of the target users and context of use. Then, according to the preliminary evaluation report and subjects suggestions, we will update the prototype to the final version of our product. Eventually, a formal evaluation maybe held in the course TIE System Design, all participants would be the students and/or research assistants of SoC design. The result of final evaluation report will present whether our product meets the needs and requirements of the target users and context of use. (G9) - The best way is to involve the target end users in the creation of the product. If you can include the target users as much as possible and gather as much information from them as you possibly can, your product has a good chance of actually responding to existing needs instead of assumed needs. Listen to the people who you're building the product for, even when they aren't sure what they want. Observe, interview, question, try it yourself. In addition to the target users you should also go see where the product will be used (if physical) or determine the environmental and technical factors involved as accurately as possible (if intangible, i.e. software). Remember that your customer is not always the target user and could have different goals for the product! (G3) How will you validate that the design follows the basic principles of what is good usability? Expert evaluation with heuristic guidelines: - By keeping in mind the basic principles of good usability while planning it. Also we will let the end users decide if it works the way they want or if something needs to be changed. (G8) - There are plenty of different kinds of lists and methods that one can use to ensure good usability for a product. For example, Nielsen's heuristic list is a good starting point and if you just type usability testing or usability heuristics to Google, you are well on your way to finding (and hopefully following) good practices for usable design. (G3) - By following the 10 heuristic commandments during the design phase. (G7) - We are going to use Nielsen's 10 heuristics to validate the usability of our product (G5) - After we build up the prototype of our product, a preliminary evaluation plan will be involved in the procedure of our project. For the purposes of evaluation of UI, we will seek help from 1-3 experienced participants. All subjects are requested as a professor/researcher in computer system in TUT. They will evaluate ten aspects of our prototype from a user perspective (professional FPGA/SoC designer) as visibility of system status, matching between system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, aesthetic and minimalist design, helping users recognize, diagnose, and recover from errors, help and documentation. If all these are satisfied by our subjects, which means our design basically

10 follows the basic principles of what is good usability. Then, according to the preliminary evaluation report and subjects suggestions, we will update the prototype to the final version of our product. Eventually, a formal evaluation maybe held in the course TIE System Design, all participants would be the students and/or research assistants of SoC design. The result of final evaluation report will present whether our design follows the basic principles of what is good usability. (G9) - We use Nielsen's heuristics and Norman'n design principles where possible. (G2) - We design a few different use cases and determine some requirements for them. For example, some of the most common use cases must be performed with just a couple of clicks (G4) Customer s takes care of UX: - Our customer is an ux-specialist and we build ui s iterative with them and they give feedback immediately to us (G10) UX specialist within team: - Our usability expert is responsible for continuously ensuring the usability. (G1) - Our UX/UI expert will go through the design and yell at us if he sees that something is wrong. Also our customer has some UI/UX experts and we might ask for their opinions. (G6) What kind of user testing or usability/ux evaluation are you planning to conduct? Eat your own dog food - The idea was to invite random people to try out the product, but due to unforeseen circumstances, that might not be possible anymore. If nothing else, all the members of the development team as well as a representative of the customer are going to test the product individually. (G3) - We use Nielsen s heuristics in design that give some good advice for the ui design. Also we use Android s design principles for the mobile-app UI. At the moment we are testing apps just by ourselves and with customer (G10) Random other people / customer: - We will ask some of our friends to try out the nearly-finished product and ask them to do some test tasks. Then also make kind of questionary about the UI for the testers. (G2) - Our plan is to do peer reviewing and user testing with the customer (G1) - Our clients are experienced users of Flowdock and Pivotal Tracker and Google Docs. They test our application mainly with exploratory testing and see if it gives any new value in comparison with using the 3 web applications separately (G4) Usability tests with real end users:

11 - we are planning to perform heuristic evaluations and usability tests with users coming from targeted users' groups (G7) - Actual users are going to test the product from early versions in actual environment. (G5) - After we build up the prototype of our product, a preliminary evaluation plan will be involved in the procedure of our project. For the purposes of evaluation of UX, we will seek help from 1-3 experienced participants. All participated subjects in the evaluation are requested as a professor/researcher in computer system in TUT. They will evaluate the categorized quality of our prototype from a user perspective (professional FPGA/SoC designer) as functionality, reliability, usability, efficiency, maintainability and portability. If all these quality are satisfied by our subjects, which means our prototype basically meets the needs and requirements of the target users and context of use; and our design follows the basic principles of what is good usability. Then, according to the preliminary evaluation report and subjects suggestions, we will update the prototype to the final version of our product. Secondly, a formal evaluation maybe held in the course TIE System Design, all participants would be the students and/or research assistants of SoC design. The result of formal evaluation report will present to the customer. Eventually, there will be a customer s testing/evaluation which depend on customer s desire. (G9) - Usability testing combined with interviews with the end users (G8) - We have talked about doing some tests with end users who do not know how the application works before the tests. We would tell them to do some things that one can do with the application and follow how he or she is able to do that without or with small amount of guiding. (G6) What are the most critical tasks or use cases in your product that you want to test with end users? - Is the product even feasible or useful? We are venturing in somewhat uncharted territory and so far we've had quite a few setbacks. We're still pushing the product forwards, but the fact is, it might end up as a technology demo and nothing else. (G3) - Is an app UI clear enough and simple to use? (G10) - Can older users use touchcreen/mobile-device? (G10) - Can a PoC application bring an advantage or disadvantage for the end users? (G10) - Selection of a specific model diagrams within many others. (G7) - Selection of many items (classes) of one model diagram (Class diagram) and read the specific description of that element. (G7) - Selection of many items (classes) of the same model diagrams and it is able to read relations between those items. (G7) - Creation of the issue tracking/improvement report has to be easy and quick because they might be created in the middle of other work at production site. Responsible persons have to find easily and be aware of open reports assigned to them. (G5) - Target game genre is constructive (like Minecraft). (G9) - 3D metaphors/visualization of SoC design objects. (G9) - Storage of building blocks in a visual library. (G9) - Simple attributes associated to the blocks telling compatibility to each other. There are taken from the IPXACT standard. (G9) - Dragdrop assembly and manipulation of the objects representing SoC design. (G9)

12 - Hierarchy by nested objects and controlled transparency. (G9) - Semantic zoom to move smoothly between 3D hierarchies and between 3D, 2D and textual views. The idea is to open a code editor when zoomed in on the bottom of hierarchy. (G9) - The gameplay is basically simple: take blocks from the store and assemble/disassemble them. (G9) - Creation of new blocks from scratch is optional. (G9) - In our home automation system the most critical tasks are probably adding the device, adding a rule, and modifying the dashboard. (G2) - Our sync station must sync the activity trackers' data to the web service successfully. (G1) - Categorizing job applications, adding notes and tags to them, searching them and answering to job application s through our application (G8) - Overall navigating in the system and finding the charts that are interesting. Also getting the needed type of data out of the charts should be tested with end users since there is some options of how to filter the data and what kind of data is used to calculate the chart values. (G6) - A customer of our client is given an URL to our application about their project. Opening this URL shows relevant data about their project after the user has logged in. After this, the user must be able to see all data about the project with 1-2 clicks (G4)

13 Questions from the groups Themes: 1. UX activities in uncommon circumstances 2. What kind of UX testing should we do? 3. Iterative design & Lean UX + bonus UX activities in uncommon circumstances - How do you involve usability/ux work in a project, where just about everything that's required is hardcore programming and the finished product is, for example, only required to basically show a picture and some lines of text? (G3) (Google Glass case) o think about the novelties in the interaction (speech interaction, touch input, AR) à what else could you do than what was asked? o tutorials & panic button o be at least customer-centered: aim at truly understanding their needs o how do you define what people want to see about the license plate? o nice UI is a relatively easy way to create a wow effect and helps selling your ideas - Our Product is not planned to become a "real product" since it is more like a technology demo. How important it is to polish the user interface? (G2) o as above o not that high priority o what feedback does the user want/need? - What if the client/end user wants something for the product that goes against the principles of good usability? Should we just go with the client or at least try to explain why it might not be a good idea? (G8) o give a few alternative ways and demonstrate how thing work with them à explain what changes (e.g. in the architecture) will it cause o examples from well-known good designs o compromise if necessary o provide reasons for your viewpoint and require that also from the customer o remember that some universal usability principles might not actually be universal à consider if this particular context of use or user group has special needs that diverge from what is normally considered good - How do you perform UX testing when you simply can't get a clear idea of your target user group? (G3) o pick a heterogeneous group of users à cover more ground o draw a line somewhere, probably the user group is not the whole world o use your product yourself and ask some other people to use it as well à at least the superficial usability can be made better

14 o usability heuristics o consider design guidelines for particular domains and application areas (e.g. mobile UI design, particular culture) What kind of UX testing should we do? - When is the optimal time to start usability test or heuristic evaluation of a product (knowing that the actual working prototype could be available late)? (G7) o asap. paper prototypes to start with & simple tests o test one module / feature at a time o optimal depends on the case. However, remember that you can test many usability aspects already with quite simple prototypes and UI sketches. Try to identify which quality aspects require a functional proto to be tested in a reliable manner, and which can be tested already earlier - Our main product is a sync device that is basically a Raspberry pi that automatically syncs the activity data from Fitbit activity trackers to Fitbit web service when the devices come near it. Also, our data display has very little user interaction because it just displays the data. What kind of UX testing could we do? (G1) o sounds like long-term testing would be needed. Consider Wizard of Oz or other way of simulating the functionality - What should be taken into account in UX testing for a single page web app? (G4) o consider the target user group and context of use (mobile vs. desktop) o It s like a landing page, which is often the most important page of the UI! Where does the attention focus first, understandability, how well it motivates to read the content, how people understand the purpose of the system, aesthetics, many other things - If the product will have only 3-4 end users, are interviews combined with usability tests enough to ensure good usability? (G8) o Based on this info, sounds like a good plan. Do they all have similar needs and habits? o Consider also expandability to other users. Iterative design & Lean UX: - What would be the best way to make continuous usability testing for the UI? (G10) o design test cases beforehand o run user tests in every sprint o utilize multiple different methods: heuristic evaluation every now and then + usability tests in 1-2 iterations + personally testing the product - Can automated GUI tests improve the UX? (G10)

15 o Please specify o may help designers with poor UX skills - Some of our end users are in Helsinki, and there might be a possibility to also test the usability with them. Are there methods for remote user experience testing? If there is, which of those methods works well? (G6) o remote paper prototyping apps do exist o video conferencing works well o Onnibus J - How do you keep design and implement reasonable, unified and constant when the requirements constantly change, even drastically? (G3) o how are they changing? small details or more broadly about what features there are? o prioritization of the given requirements, are they really needed. Tell them what would the changing mean (more costs etc.) o revising the requirements and trying to freeze them - The customer wants to test an early version of the product in actual use. How should we prepare ourselves and the customer to get the most UX and usability information/feedback from the testing? (G5) o think what can you actually test with the current functionality Bonus: - In our project, the model with single-color and same shape represents a category of components in SoC designs, therefore colors are needed to use in our design, which will cause visual conflict and mess. This situation should be avoided in UI design. Can we decrease the usage of colors without affecting the implementation of components? And how? (G9) - Should color-deficient (color blind) situation be considered in our design? Should the psychology of color and gender be considered in our design as well? (G9)