Usability evaluation in practice: the OHIM Case David García Dorvau, Nikos Sourmelakis coppersony@hotmail.com, nikos.sourmelakis@gmail.com External consultants at the Office for Harmonization in the Internal Market (OHIM). Alicante, Spain 1 Usability evalution in practice: the OHIM Case
Index 1.Introduction 2.Methodology 3.Findings 4.Validation 5.Threads to the validity of the results 6.Future steps 7.Conclusions 2 Usability evalution in practice: the OHIM Case
1. Introduction, project background What is the OHIM? The purpose of the Office for Harmonization in the Internal Market (OHIM) is to register, promote and administer trade marks and designs at European Union level. The OHIM is an independent, public, European establishment that is legally, administratively and financially independent. Its activities are governed by European s Community law. OHIM is based in Alicante, Spain and has approximately 1000 employees (Internal and external staff). 3 Usability evalution in practice: the OHIM Case
1. Introduction, project background The need for a redefined OHIM Web Site arised from a study carried out during the fall of 2005. Major findings: The OHIM web site: was not easy to use and learn did not meet user expectations had a bad and inconsistent user interface that created frustration and problems. Therefore on October 2006 the Management Commitee approved the launching of the OHIM website 2008 project 4 Usability evalution in practice: the OHIM Case
Methodology Define hypotheses Define user profiles, chose sample population and testing enviroment Design the test Chose the artifacts on which the test is going to take place Execute test and gather results Analysis and report Actions taken 5 Usability evalution in practice: the OHIM Case
2. Methodology: Usability Hypothesis Navigation - The users can navigate with ease through the new web site. Interactivity/Structure - The users can understand the structure/labelling and organisation of the information on the web site and interact with ease. OHIM s new web site generates a positive attitude towards both the office and the web site. Based on the analysis of the findings the first and the third hypothesis were proven. The second one was not. 6 Usability evalution in practice: the OHIM Case
2. Methodology: User profiles General public. People that do not know about intellectual property (IP) and OHIM Paralegals. Users representing people that use the e-business tools and OHIM web site frequently and have knowledge of intellectual property. Lawyers. Users with knowledge in IP that are already using our web site. 7 Usability evalution in practice: the OHIM Case
2. Methodology: Usability lab A meeting room converted as a usability lab 8 Usability evalution in practice: the OHIM Case
2. Methodology: testing artifact An incomplete version of the new web site was tested. Around 40% of the total pages were available although all the pages related to the scenarios were complete. The site is designed for resolutions of 1024x768 pixels and higher. Therefore the resolution of the users screen was setup to 1024 x 768 pixels. 9 Usability evalution in practice: the OHIM Case
2. Methodology: Metrics The usability team defined the following usability metrics Code Pass/Fail Menu label Description 1 Pass Easy 1st try - no problem 2 Pass Medium 2 nd /3 rd try- observed difficulty 3 Fail Hard 3 rd /4 th try- expressed difficulty 4 Fail Assist Succeeded with assistance 5 Fail Fail Failed or gave up Relative to users performance on a given set of tasks (Task scenarios were grouped by complexity) 10 Usability evalution in practice: the OHIM Case
2. Methodology: test procedure The participants were requested to: Answer pre-test questions Complete a set of task scenarios (between 10-12 scenarios depending on the profile). For lawyers and paralegal groups 66% of scenarios were the same. Answer post-test questions Complete a standard post-survey called System Usability Scale (SUS) 11 Usability evalution in practice: the OHIM Case
2. Methodology: test procedure Two pilots were contacted for fine tuning the test The think aloud method was used The scenarios were created along with the project owner and based on the business goals The Usability Data logger Tool was used to log notes and analyse the results (www.userfocus.co.uk). 12 Usability evalution in practice: the OHIM Case
2. Methodology: analysis and report The analysis of the findings and the report was completed in 2 weeks. Various meetings held with the project owner prior finalizing the report The report layout was partially based on a standard usability report template (www.usability.gov) The report focused on findings, recommendations and actions to be taken. As the reports target audience were OHIM s managers. The recorded sessions on video was used as a backup to notes and for reviewing when the usability team was disagreeing. 13 Usability evalution in practice: the OHIM Case
3. Findings: categorisation Type of nature: Graphical Structure Navigation Labelling Users satisfaction Severity: Low Medium High Complexity: Easy Medium Difficult The categorisation by severity was defined by the usability team based on the business goals set by the project owner at the beginning of the process The categorisation by complexity was defined by the usability team after talking to each responsible of fixing the problem/bug 14 Usability evalution in practice: the OHIM Case
3. Findings: positive findings Finding Contextual menu Easy to use and navigation is obvious and in general understandable News section on the home page Site design mainly for Pro users but also helps the non-pro s General satisfaction based on SUS Uncluttered and consistent layout Overall layout and look & feel Category Navigation Navigation Structure Structure Users Satisfaction Graphical/Visual Graphical/Visual 15 Usability evalution in practice: the OHIM Case
3. Findings: positive findings High general satisfaction (average of 81.5 out of 100) Paralegals scored 88.1, Lawyers 83 and General Public 73.5 Q P1 P2 P3 P4 P5 AvgSUS 1 1 1 2 0 1 2 4 4 3 2 4 3 3 4 3 1 3 4 4 4 4 1 3 5 4 3 2 3 2 6 4 4 3 3 4 7 2 4 2 2 4 8 4 4 3 1 3 9 3 3 3 3 3 10 4 4 3 4 4 SUS 82.5 87.5 70 50 77.5 73.5 Characterisation of the web site: Service Oriented Informative Good Friendly Stable SUS satisfaction results for general public: 16 Usability evalution in practice: the OHIM Case
3. Findings: negative findings Finding Title Category Severity Complexity Small grey header, top of page Graphical/Visual High Medium OHIM and You section Structure High Difficult About OHIM section Structure High Difficult OHIM Contact info Graphica and Structure High Easy Application pages Graphical and Structure High Easy EUROLOCARNO Labelling Medium Easy Publications (CTM, RCD, Newsletter and OHIM) Structure Medium Medium Basics pop up Navigation Medium Easy IP offices and courts contact info Structure Medium Medium Customer care Structure Medium Medium MyPage Structure Medium Medium Networks and related websites Labelling Medium Medium Service charter Labelling/Structure Medium Easy The registration process Structure Low Medium Forum Structure Low Easy Institutional Info Structure Low Easy 17 Usability evalution in practice: the OHIM Case
3. Findings: an example Example of finding: small grey header, top of page Findings and users comments The grey header exists at the very top of each page and has on it the language drop down list, font size change icons, the FAQ, the contact and the site map link. A severe visibility problem was identified here. Scenarios regarding the contact info had a very high rate of failure (only 2 out of 14 finished the scenario successfully). One user (general public profile) found it by chance while he was doing another scenario and he failed to find it on the scenario that was actually designed for that purpose. Almost all the users tended to scroll and hide the top grey header throughout the test process as if it did not exist. One user asked at the end of the test if this grey header was part of the site or the browser. Two users asked if the site was only available in English (they could not see the language drop down menu on the grey header). There were users looking for 2 and 3 minutes at the page without being able to see the information that they were looking for on the grey top header! Most of the users after finishing the tests asked us to show them where the contact information was located. All of them were surprised and most of them commented that they did not notice the grey header and its content at all! Recommendations It is obvious that the colour of the grey header has to change in order to make it visible and not confused with the colour of the browser. A shade of blue can be used in order to be in line with the main blue header. Action taken A design revision has been implemented with the bar of information coloured blue. It has a discreet separation from the OHIM title panel. The word contact has been changed to contact us. 18 Usability evalution in practice: the OHIM Case
3. Findings: action taken BEFORE NOW 19 Usability evalution in practice: the OHIM Case
4. Validation In order to validate the usability test design, we discussed the approach with a critical audience which helped us to identify problems and weaknesses, challenged lack of clarity and suggested modifications. External usability experts validated all the test material and process External usability experts validate the analysis and report Validated against existing usability methodologies 20 Usability evalution in practice: the OHIM Case
5. Threads to the validity of the results An incomplete version of the new web site was tested (around 40% of the total pages were available). There is a possibility that the scenarios could be completed via alternative paths if the whole site was ready. to minimise the risk all possible paths to complete the scenario were available although this cannot be compared to a full operational site. Hypotheses were formulated addressing to the usability of the site as a whole. It was not feasible to cover all the aspects of the web site in the task scenarios (due to time restrictions, not complete web site etc.) Therefore there is a possibility that the general findings of the whole site to be slightly different than the ones extracted from this research. 21 Usability evalution in practice: the OHIM Case
6. Future steps The OHIM is planning a new evaluation of the web site, in the form of contextual interviews, surveys etc. It will help OHIM to identify if the actions taken were proper and compare user satisfaction with earlier research findings. 22 Usability evalution in practice: the OHIM Case
7. Conclusions OHIM realised the usefulness of usability testing and is willing to conduct more tests in the future for various products (various applications, intranet, etc). A decision to shift to a more user centered design process for development of IT projects has not be taken yet as it requires a lot of time and major changes in OHIM s processes and organisation. 23 Usability evalution in practice: the OHIM Case
THANK YOU! David García Dorvau, Nikos Sourmelakis coppersony@hotmail.com, nikos.sourmelakis@gmail.com External consultants at the Office for Harmonization in the Internal Market (OHIM). Alicante, Spain 24 Usability evalution in practice: the OHIM Case