This PDF was generated from the Evaluate section of

Transcription

1 Toolkit home What is inclusive design? Why do inclusive design? How to design inclusively Overview Map of key activities Manage This PDF was generated from the Evaluate section of This toolkit was developed by the University of Cambridge, Engineering Design Centre, sponsored by BT. Explore Create Evaluate Further instructions Inclusive design tools User capabilities Case studies Links and resources If you would like to purchase printed versions of this toolkit, please contact for details. Page 1 PDF created: 19/03/2015

2 Evaluate This page describes activities within the Evaluate phase of the design cycle. These activities help you to evaluate the concepts produced in the Create phase. They examine how well the product criteria are met, considering the needs of all the stakeholders and target users. On this page: Review criteria Test with experts Test with users Estimate exclusion Present evidence Page 2 PDF created: 19/03/2015

3 Review criteria It is important to set criteria for determining if a product is successful. These should be based on the needs that were identified in the Explore phase of the design cycle. The criteria will typically cover issues to do with People, Profit and Planet. People issues include: User experience, Total cost of ownership and Social impact. Profit addresses: Costs and revenues, Technical risk and Business risk. Planet covers: Depletion of scarce resources, Energy use and Waste impacts. These criteria are described in more detail in the Integrated design log). These criteria have some relevance to any project. However, their relative importance may vary. At this stage in the design cycle, it is important to review the criteria and discuss which are the most important: deal-breakers or unique selling points. At this point, it is also important to plan how the criteria will be tested. Start by choosing a benchmark to test against. This is helpful becuase it is much easier to judge whether something is 'better' or 'worse' than something else, than to make an objective assessment of whether something is 'good' or 'bad'. The benchmark could be an existing product, or an existing way of accomplishing a similar task. It is also important to plan how the various criteria will be measured. For example, user experience may be measured using time taken, error rate and satisfaction during a user test. Headline criteria can also be used on their own for quick and initial evaluation during the early prioritisation of ideas and concepts. You can download an Integrated design log that contains a blank template to help you agree criteria. Page 3 PDF created: 19/03/2015

4 Test with experts In this activity, a range of relevant experts use their skill and knowledge to systematically judge and test concepts against the agreed criteria. Expert judgement is needed because it may be difficult to formally test how well initial concepts could ultimately perform against the different criteria. A multi-disciplinary team is needed in order to make judgements against all the different criteria. This ensures the complete set of criteria is appropriately represented and prioritised. The initial descriptions of user journeys produced in the Explore phase can be extended into a comprehensive task analysis in order to provide a framework for the expert appraisal. This task analysis should cover all aspects of the use of the product, including purchase, installation, use, maintenance and disposal. For each task step the concept's performance should be evaluated against the relevant criteria. The understanding of users gained from the Explore phase can also be used to help evaluate the concepts from different user perspectives. The comprehensive task analysis and the insight gained from expert appraisal should be used to set the scope for subsequent evaluation activites. You can download an Integrated design log that contains a blank template to help you test with experts. The capability simulation tools available on this site can also assist with expert appraisals, helping experts to understand how capability loss may affect product use. These include the Cambridge simulation gloves and glasses and the Impairment simulator software. Further information James Hom's Usability Methods Toolbox website describes various "inspection" methods that can be used by experts to help them inspect and evaluate a concept or product in a systematic way. The Usability Net website has a section on Heuristic evaluation. This is a particularly popular method of expert appraisal, in which the product is evaluated against established guidelines or principles. Page 4 PDF created: 19/03/2015

5 Test with users Testing with users evaluates whether they can use product concepts and prototypes, and how much they like using them. It can identify sources of frustration and difficulty so that they can be addressed. Early user testing is vital because it is hard for designers and experts to correctly imagine the details of what users will do. User testing can take place at different levels of formality and detail. Early in a project, it is often informal and qualitative, providing initial feedback on concepts and ideas. Later on, it can be used with more objective measures, such as timings and formalised questionnaires, to ensure that products fulfill criteria and standards. The understanding of users and stakeholders, developed within the Explore phase, can help to choose who should be recruited for the user tests. The task analysis produced as part of 'Testing with experts' can help to identify the tasks they should perform. You can download an Integrated design log that contains a blank template to help you test with users. Further information The Designing with people website has sections providing further information on User methods and the Ethics of involving users. James Hom's Usability Methods Toolbox website has a section on Usability testing which provides more information on objective user tests. The Usability Net website has an entry on Performance testing which also gives more information about usability testing. Page 5 PDF created: 19/03/2015

6 Estimate exclusion It can be helpful to identify who would be excluded from using a product or prototype on the basis of their capabilities, such as: Vision, Hearing, Thinking, Reach & Dexterity, and Mobility. This can help in evaluating the accessibility of a product for a diverse range of users. To do this, it is helpful to work through the task steps involved in using the product (such as those identified in the task analysis produced during 'Testing with experts'). For each step, the assessor examines the demands placed by that step on the various user capabilities. Reducing these capability demands (while achieving the same features or functions) should lead to a more satisfying product that can be used by a wider percentage of the population. Estimating exclusion should not be used in isolation but should be complemented by the other activities in the Evaluate phase of the design cycle. In particular, user testing helps the assessor understand how to score the demand levels of each task. You can download an Integrated design log that contains a blank template to help you perform an initial estimation of exclusion. This toolkit also contains an Exclusion calculator. This allows you to estimate the demands that a particular task places on the user's capabilities, and then calculates the proportion of the UK population who would be excluded by that task. Further information Keates and Clarkson (2003)'s book 'Countering design exclusion: An introduction to inclusive design' provides more background on exclusion calculations. (Published by Springer) The Inclusive Design website, produced by the Cambridge Engineering Design Centre is an online tutorial that accompanies Keates and Clarkson's book and provides some background to exclusion calculations. A paper by Goodman-Deane et al. (2011): 'Estimating exclusion: a tool to help designers' gives more recent information on exclusion calculations. (Published in Proceedings of Include 2011). Page 6 PDF created: 19/03/2015

7 Present evidence This activity draws together, summarises and communicates all of the evidence that has been generated from the evaluation activities. To do this in a systematic way, it is helpful to look back at the evaluation criteria agreed on in 'Review criteria', and document the evidence obtained in each of the key areas, such as User experience, Social impact and Costs and revenues. Doing this highlights if evidence is missing in some areas, and can help to determine if more evaluation is needed. The evidence in each area can then be compared with the criteria to drive the objective choice of the lead concept to focus on in the next cycle of concept design. You can download an Integrated design log that contains a blank grid that allows you to summarise the evidence gathered according to the criterion it addresses and the concept being tested. Page 7 PDF created: 19/03/2015