TIE Project Work on Pervasive Systems

Transcription

1 TIE Project Work on Pervasive Systems Workshop on UX evaluation, Thomas Olsson Groups answers to warm-up questions How do you validate that the product matches the needs and requirements of the target user and the context of use? Customer as users voice: - First of all we meet the customer often enough to make sure we have the same view about the finished product. Then we show our UI and functionality in our customer meetings to help them ask the questions and to notify us if something is not what they expected. In the testing phase we give the current release/demo and provide a feedback sheet where they can put their thoughts on and ask their questions. This is how we have worked this far and will continue as soon as the testing phase really starts in January. (G4) - Requirement validation is basically handled in our customer meetings by customer acceptance, since our project isn't going into production use for certain group of people - results are used as a technique research for the customer. Some sort of use case scenario testing is going to be implemented based on customer requirements, which should give the needed feedback for our group. (G5) - Present the application to the customer and customer presents it to the users. From the feedback we validate if the product matches the needs. (G10) - Include customer to the testing process. (G8) Users involved: - Allow the end user to test the product. Monitor them testing and make notes. (G6) - Our product is a massively multiplayer browser game, so we are planning to start a closed test to find possible defects and receive feedback. The testing should start in two weeks time. (G2) - By testing UX... The tests also include a small interview/survey before and after the test session (G3) - With at least one usability test and using heuristics. And of course with design, that starts from understanding of our users and their needs. (G9) - Plan is to conduct a user study of the implemented product and focus in validating that the system is perceived to be as easy to use as the old system. Second focus is that the system provides tool management functions that match the requirements that has been gathered. (G7)

2 - Software quality can be measured internally (by static measures of the code), or externally (by measuring the behaviour of the code when executed). E.g. reliability can be measured externally by observing the number of failures in a given period of execution time during a trial of the software, and internally by inspecting the detailed specifications and source code to assess the level of fault tolerance. For the product to have the required effect in a particular context of use. Quality in use is the user s view of quality. External properties (such as suitability, accuracy, fault tolerance or time behaviour) will influence the observed quality in use. A failure in quality in use (e.g. the user cannot complete the task) could be traced to external quality (e.g. suitability or operability) and the associated internal attributes which have to be changed. Measures are normally required at all three levels, as meeting criteria for internal measures is not usually sufficient to ensure achievement of criteria for external measures, and meeting criteria for external measures is not usually sufficient to ensure achieving criteria for quality in use. (G12) What kind of user testing or usability/ux evaluation are you planning to conduct? No plans: - Our project doesn't have actual UX-requirements (besides "UI should resemble the old UI") and our customer has announced that UX-matters aren't really in the project scope, so no actual user testing isn't planned. (G5) With the help of customer: - We are not sure yet whether we actually do some kind of pre-constructed sheet to evaluate our UI based on testing but what we are doing for sure is that we give the tool to our customer for testing purposes. We provide a sheet with questions how they would evaluate some particular things in the UI and such and then the open feedback area as well. We might use some third party testing group, but there are no plans about that yet. (G4) - We are user-oriented. we hold weekly meeting, where we give demo on our prototype. We use questionnaires in the beginning of design, with a storyboard, we easily review the whole flow of the user interface and interactions. during product development, our customers review our UI that they wanted to implement. (G12) - Show software to the customer in every demo session. No specific user testing is planned. (G10) Usability test sessions: - Sessions with test users using our system, doing predetermined tasks. The sessions will be recorded with a video camera. (G3) - The testing mentioned above. (G2) - The group will take care of the basic usability testing. No concrete plans for customer executed testing have been made. (G6)

3 - Heuristics and user tests. Also material from previous usability studies on earlier iteration of the product available. (G9) - We are planning to install test server to our customer, so they can test program as it develops. Also, we have our own UX- specialist who will do some testing. UX- specialist will also do some interviews to people outside of the project to get more data. (G8) - We have used generally approved elements (bootstrap and android development style guides) in our system and especially UI development to ensure that we get a good level of usability from well thought out guidelines and frameworks. During the planned user test, observations regarding ease of use will be analyzed to be used in the next iteration if needed. (G7) What are the most critical tasks or use cases in your product that you want to test with end users? Everything! : - The whole experience should be as smooth as possible, so there isn't a specific task or use case we are going to test. Administrative tools are not as important, as only game staff is going to access them, so they are not so critical. (G5) Specific functionalities: - Creation of a simple survey, creation of a multibranch survey, getting analyzed data, creation of own themes for surveys with CSS-editor, adding and deleting access rights in the admin mode (G3) - Definitely the ease of editing your own profile: adding new skills, set the values for a skill, join to a project, leave from a project, delete a skill from your profile... basically all the biggest relations between User - Skill - Project. The other main functionalities in addition to these are search function and authentication. (G4) - We would like to monitor where a first time user searches for functionality and analyse the differences in their expectations and our decisions. (G6) - Simple enough for our users, which are epilepsy patients. Inputting the seizure entry is the most critical task. (G10) - Data must be correct all the time. Our program will include some automatic calculation and data is quite important for business, so it is important that automatic operations work as user expects (G8) - 1) User loans a tool from a static cabinet, 2) User returns a tool to a static cabinet, 3) User loans a tool from an other off-site using mobile internet connection, 4) Manager checks which tools are in need of calibration or repair, 5) Manager checks which holder has a specific tool (G7)

4 Other: - Again, no end user testing is going to be arranged, but the most critical tasks would be HTML5-functionality and solving its adequacy for the required functions in our project (canvas-drawing for example). (G5) - we are doing fleet management app, using vacuum cleaning robots are fleets. one of critical use case is how to get a map for cleaning robots to follow. As robots learns about environment from the its sensors, our customer want to get a map from robot movement, once it detect walls or obstables, it will print the location on the map. after robots move around the whole room, a map should be generated. (G12) - NDA, sorry. (G9)

5 Questions from the groups Theme 1: Users are nasty and difficult to find: - how to test UI functionality with an audience that is accustomed for a certain way of doing things and are expecting things to work as "always"? (G5) o A-B comparison with the old system or just using the old system as baseline o As they might focus on too small details in the UI, you might first need a short introduction of the new system and its parts (compared to the previous one) o To overcome the resistance to change you need to 1) convince some people in the organization already beforehands, 2) clearly communicate the expected benefits ( the aim is to make this more user-friendly and make it more suitable especially to YOU ) - Our customer doesn't want to provide us with any rewards to give to the test users and nor are the project members willing to spend their own money. How can we find participants? (G3) o Other groups of the course (win-win)? students from the hallways? do the user representatives need to know about the target domain/tasks? o Apart from monetary incentives, people have also other motivations to participate in testing and science in general - We have a set of predetermined tasks for the test user. How much should we help the user when he/she faces difficulties and bombards us with questions in the middle of the session? What if he/she is lazy and doesn't even bother trying to complete the tasks properly? (G3) o The questions are probably good feedback about what s/he didn t understand about the system. If they re irrelevant, just gently move the focus to the system o Laziness is usually not a problem. In the beginning, it s good to introduce the overall approach of the testing ( please complete the given tasks as if they were your actual duties instead of just use the system and tell us what you like ) - How to do usability testing, when the real user group is hard to get? (G10) o Do the user representatives need to know about the target domain/tasks? You can test some objective elements of usability also with just anyone o Consider also expert analysis o Consider remote methods like online surveys for easier reach of the target users - We are planning to interview people about out projects usability. How should we handle situations, where people have really mixed opinions? (G8) o First of all, try to understand why they think as they do o Prioritize according to who are the main users or which are the main features o Try to involve the customer in making the decision, if they happen to be active in the project - How many users should we recruit for usability tests? (G1) o For the purposes of this course, 2-4 should be just fine and reveal the most critical issues

6 Theme 2: Methods & measurement: - Are there any other ways to test usability than letting users use the system and ask their opinion and monitor their way to use the program? (G6) o Expert evaluations based on heuristics and walkthroughs with stakeholders o Guerilla tests & hallway testing o In group: go through the main use cases with as fresh eyes as possible and e.g. count the number of interactions you need to complete a task o role playing in evaluation o Comparing your solutions with existing software that you think are usable - How should we test the usability of a game? We know how to test a normal interface, but testing seems daunting with our levels of complexity. (G2) o Usability is naturally an important hygiene factor of games. However, usually the UIs are rather simple and therefore rather easy to design to be usable, so basically what matters more is the UX: fun, stimulation, engagement, intention to play again, what kind of feelings and emotions does it create - What usability testing methods would be best for after integration phase usability testing when there is about 2 weeks time left? Focus of tests would be in comparing old and new system. (G7) o This sounds mainly summative evaluation: getting estimates of the overall quality with some numerical measures (overall ratings). Small fixes can probably still be implemented so it s good to also gather users proposals for improvements and identify problematic parts (e.g. terminology, placement of UI elements, o To consider: gathering good ideas for further development is not only valuable for your customer but might also help you get employed to the company - What testing method could be used for evaluating user experience? How about without test users? (G7) o The basic structure is more or less the same as in a usability study but the measures are often more broader o Try e.g. AttrakDiff, sentence completions and focusing on the user s feelings and reactions in the interview - How to measure usability? (G10) o Various existing metrics exist: SUS, SUMI, QUIS, etc. These also allow for quantifying the results à numerical data and possibility for comparisons. o (The more objective and generalizable data you want, the more controlled the setting needs to be à get familiar with experimental research (control variables, counterbalancing etc.)) - What do you suggest on how to construct the feedback sheet used on UI UAT testing phase? Functionalities listed which should be evaluated? Just a blank paper? (G4) - What evaluation methods could (your, or common opinion :) be listed as "must-do" in all ITprojects? (some examples) (G5)

7 o It s not about which methods to use but it s about 1) ensuring that the product answers some actual end user needs (not just your or customers assumptions about them) and 2) refining the workflow & usability of the most critical use cases / functions with a lo-fi prototype. At least both these phases should be considered but which methods to use in doing that is another question. In that regard, at least you should do expert evaluations, either within your team or with external experts. Regarding user involvement: the earlier, the better. - What needs to be taken into account in usability testing in the case of a mobile application? (G1) o Test it in a mobile environment where it is intended to be used, e.g. in public transport or a café; allow interruptions, noise and other disturbances that often affect use of technology in mobile contexts o consider different display sizes as well as input techniques - What methods to use? (G12) Which measures/metrics/heuristics to use in the evaluation? (G12) Theme 3: Testing with minimum effort/resources: - How to find time and resources for user testing because the coding takes all the time; in other words, how to test with minimum effort (G5) o Narrow down the functionalities you re going to implement. Of course it s an internal decision (or customer s) whether you do as much as possible with minimum quality or if you do a narrower set of features but with high quality. Testing in general is all about enhancing quality. Have you defined your targets of the project well enough and is there a consensus about them? o Consider expert evaluations and the fact that you don t actually need that many users to find you most of the critical usability issues. o Hallway-testing & guerilla testing o Testing communities online where people participate just to help each other s projects o At least check out possible design guidelines and styleguides - If there are two alternative approaches to improve usability, how should one determine which to use? Let s assume there is no time/resources to implement both. (G6) o So what are the criteria for selecting the methods? First you need to understand what you actually want to achieve with the user testing (summative vs. formative testing, qualitative vs. quantitative, subjective vs. objective). E.g. how reliable and credible results do you need? Then you can think about which method to use and how much (if at all) you can involve end users in the process. Different methods have different strengths and weaknesses. E.g. involving just someone as users might be much less useful than doing an expert evaluation with a list of heuristics. On the other hand, only the actual end users can determine if the system is suitable for their context of use and their needs. - At the moment it seems that we will be only able to conduct one user test. We will use to test our prototype. How to test with this amount of tests available? (G9)

8 o Make sure you get an honest and analytical (and talkative) user representative. Focus on the most critical tasks / use cases, and only after them aim for coverage of the system. o You cannot conclude anything general based on one user so it s best to focus on formative evaluation. - How wide target group we should choose for interviews? It might not be reasonable to interview 60 year old factory worker for business intelligence program or is it? Should we focus only for target users or interview large groups? (G8) o For evaluation purposes on this course, just select 2-4 representatives of the most relevant target groups; in the requirements gathering phase, it can be useful to gather insights from especially this kind of very experienced persons Theme 4: Project management, resource allocation, communicating the results: - How much time should we save for fixing the found problems? Implementation of promised features should not suffer, but then again the product should be as good as possible. (G2) o In a project with fixed schedule (like this) perhaps it s best that you just schedule predefined amount of resources for fixing them (e.g. a week from one guy, or 1-2 days from all the coders). o In any way it s good to prioritize the evaluation results (e.g. criticality, or how much time they require to fix) and then start from the most critical or easiest ones. o First make sure that the software functions right, only after that focus on how it looks etc. - How to convince the project group to fix the found usability problems? Let's assume the rest of the team is more focused on completing the feature list and getting the credits. (G3) o Project manager! Well set goals that are commonly agreed on! Resistance for change is very common either inside the team (having to redesign) or from the customer s end. o User quotes & personas are pretty credible way in communicating the found issues. o Consider what kind of a picture you want to give about yourselves to the customer. Becoming successful in what you do is all about getting good references and testimonials. - What if the customer/end users don t have strong enough opinions on usability? How long should opinions be attempted to be gained from them and at what point should some solution just be implemented and presented/made final? (G6) o This is really hard. Trust your design instinct, don t just listen to what the customer or the users say. They often mislead you from the great vision you have ;) In software industry (the way how I see it), many customers trust that you do the hard choices that s why they have paid you for the project. - How to visualize well the usability test conclusions about efficiency between old and new system? (G7)

9 o Comparison side by side with issues highlighted? Check how e.g. customer products are compared in online reviews. o Any quantitative measures that you can gather about both systems people (especially engineers) believe the numbers (e.g. task completion time, number of errors, subjective feeling of frustration) - What if customer has dramatically different opinions about UX than project group? Should we just do what customer says even though group does not agree or continue to develop as we see best and trust that customer will understand our way when development is further? (G8) o gather more credible data with good number of users and e.g. user quotes as evidence o at least list the unfinished features and rejected solution in the final document Other (implementation & UI design): - not really an UX-testing question, but what is a common way to implement the resizeelements in a UI? (Vertical/horizontal bars that can be dragged to resize an UI-element) (G5) o sorry, not my expertize. Don t all IDEs for desktop computing provide a ready-made feature for this?