WHITE PAPER CROWDSOURCED USABILITY TESTING SIX KEY STEPS TO HELP YOU LAUNCH AN INTUITIVE APP WITH THE SUPPORT OF CROWDSOURCED USABILITY TESTING www.applause.com Page 1 of 9
TABLE OF CONTENTS INTRODUCTION...3 The Old Way is Flawed...3 How Crowdsourcing Changes the Game...4 KEYS TO CROWDSOURCED UX TESTING SUCCESS...4 Pick the Right Crowdsourcing Company...4 Pick the Right Participants for Your Test... 5 Know What to Look For... 6 Know What Kind of Feedback You Want...7 Don t Forsake Experts...7 Don t Rush It...7 CONCLUSION...8 ABOUT APPLAUSE... 9 www.applause.com Page 2 of 9
INTRODUCTION APPLAUSE CASE STUDY ACCELLION The goal of usability testing, simply put, is to make sure that a user can complete the tasks they are expected to complete. Usability testing doesn t test whether or not the functions of the application, website or connected device work correctly, but rather that a user intuitively understands how to perform these tasks and how easy or difficult it was to do so. With usability testing, close enough won t cut it. A product may have a superior architecture, a great set of features, good performance, scalability and a number of other positive attributes. However, all of this effort is wasted if the user experience is inadequate. An application, website or connected device that is not user-friendly is just as bad as a buggy version and can lead to diminished revenue, product abandonment or a total failure. An application with poor usability can also negatively affect a brand s reputation. With the short patience of today s consumers, this is an especially dangerous risk to take you might only get one chance to make a good impression. If usability problems are not considered valid bugs, your project defines the testing task too narrowly. Testers are restricted to check whether the product does what was intended, not whether what was intended was useful. Customers do not care about the distinction. BRIAN MARICK, SOFTWARE TESTING SPECIALIST THE OLD WAY IS FLAWED Despite its importance, companies have historically neglected usability testing. The old way to test usability was expensive, time-consuming and difficult. Companies seeking to perform thorough usability testing had no choice but to send the responsibility to an outside user experience company or schedule time to bring an expert in. Because of this, usability testing often felt isolated and disrupted the QA process. On top of that, unnatural testing environments could skew results when people know they re being watched they can get nervous or self-conscious, which might affect the way they interact with a product. www.applause.com Page 3 of 9
To avoid these difficulties, companies often rely on in-house testing efforts for usability testing. This approach is flawed from the start. Internal employees are too close to the product and already know how the application works. They lack the fresh sets of eyes that external testing provides. On the other hand, friends and family who are recruited as test participants may generate biased results because they don t want to upset anybody or are concerned about not giving the right answer. Plus, neither of these options (internal team or friends and family testing) will necessarily mirror your target audience an important component of usability testing. HOW CROWDSOURCING CHANGES THE GAME The rise of crowdsourced testing has made usability testing much more accessible and flexible. Today, an informative round of user testing can be carried out in as little as 10 days, producing results that not only provide benchmarks, but also allow for immediate turnaround and implementation of suggested changes. The shortened time frame and less-disruptive process mean that companies can fit UX testing comfortably into their overall testing efforts. Since users complete the tests in their own environments and are often professional testers who are used to honestly reporting their findings, the results are more accurate. Best of all, crowdsourcing means usability testing no longer breaks the budget of startups and small businesses sectors that had historically been priced out of these services. But not all crowdsourced UX testing services are created equal. For crowdsourced usability testing to be successful, you need to keep a few key things in mind. KEYS TO CROWDSOURCED UX TESTING SUCCESS If you want to have a successful crowdsourced usability testing experience, take some time to consider these factors. Knowing what you want from the process itself and from the testing results will help you have a successful crowdsourced experience. PICK THE RIGHT CROWDSOURCING COMPANY There are several different crowdsourced testing options that offer different levels of service. If you pick one blindly you could end up with a disappointing experience and little usable feedback. Some online crowdsourced UX testing solutions make usability testing seem simple and easy, but in reality rely on people with no testing experience. In these cases you can end up with little more than noise unhelpful feedback like ugly, frustrating or useless. What exactly is ugly, frustrating or useless? The whole product or just certain features? Why do they feel this way and how can you make it better? You might never know because you might not have follow-up access to these users. It s also important to look at the features and services of the crowdsourced company to make sure they are offering everything you want or need. Important features to pay attention to are: bug-tracking system integration options, how results are presented, access to the right demographic of testers, and a level of involvement from your crowdsourcing company that matches internal needs and expectations. If you have someone in-house that can set up, manage and analyze a usability test then you might use a more hands-off company. If you want the soup-to-nuts experience, choose a company that offers more comprehensive services. Both options are available in the market today. www.applause.com Page 4 of 9
If possible, look into the service from the tester side. How big is the community? Do people seem to like testing for this service and do they keep coming back? You want to make sure you choose a crowdsourcing option that will give you access to good participants. Looking at how a company interacts with their crowd is a good indicator the more engaged a company is with its crowd, the more likely these users will stay engaged with the company. Another important factor to selecting a crowdsourced company for usability testing is continued access to the same testers. Some companies simply let testers claim projects and don t give you much control over the process other than setting demographic parameters. This leaves you in a lurch if you have any questions following a test or would like the same users to test a later version of the application. If you plan on integrating usability testing into your overall QA plan it will be helpful to have more direct access to specific participants. PICK THE RIGHT PARTICIPANTS FOR YOUR TEST As technology becomes more common place, it becomes harder to test. Gone are the days of a handful of devices, operating systems and versions we re now in a world where almost everyone has a device and the hardware/ software combinations are almost countless. This means companies have to make sure their applications, websites and connected devices perform correctly and look good on a host of different devices. Significant quality improvement is achieved when testing is done across a wide set of platforms, locations, languages, etc. Crowdsourced testing is the easiest way to cover this terrain. Testing participants can be recruited for each OS, browser, plug-in, mobile handset model, language, location and any other factor that mimics your end user base. Supposing is good, but finding out is better. MARK TWAIN Selecting participants that mirror target user demographics is more crucial to usability testing than to any other testing. If your application is designed for new mothers, a man without children isn t the ideal tester. You want participants who will use the application like your end users they re the ones who will find the UX bugs that your end users are most likely to notice or be bothered by. If your product is intended for more general consumption, be sure to include participants that represent all types of consumers. A general sports website should be usable by people who know little about sports and by dedicated fans who can handle more specific, nuanced information. If the site is too simple, the hardcore fans will turn elsewhere. Likewise, a site that is complicated and filled exclusively with sports-jargon will confuse and ultimately drive away casual fans. You ll only know you ve struck the right balance if your usability test includes both sets of users and people in between. www.applause.com Page 5 of 9
On a hardware level, pay attention to the most popular makes, models, platforms and versions among your target users. If users are more likely to be Chrome or Firefox fans, testing on Internet Explorer is less important. Does a certain mobile phone manufacturer dominate your target location? Employ testers who have models by that maker. By the time your application reaches the hands of usability testers you should already have your target market identified. Use this information to help guide testing. KNOW WHAT TO LOOK FOR A usability test gathers feedback on any aspect of the user s experience that is troublesome or could use improvement. A complete user experience covers many aspects, including: Functionality: A product or application must present the user with the appropriate functionality. If the functionality of an application is perceived as incomplete or inadequate, customers may be lost. Layout & Design: Good layout and design allow a user to easily complete tasks. If a button is placed in the perceived wrong place, users will get frustrated. If text color is too hard to read, the application will be unusable. Interaction: The flow of an application must be natural and allow the user to easily complete tasks. If a user believes they have been sent to the wrong page and have to manually backtrack in order to find the page for which they were looking, they can become frustrated and abandon the application. Language: The language used on all features of the application should be clear and understandable. Overly complicated language will confuse users and leave them feeling frustrated. Make sure all translations are correct and make sense to native speakers. More often than not, technical or inappropriate labels and terminology make it into a consumer application. Usability testers will tell you that Discounts sounds better than Adjustments or they might let you know that your information section is too verbose or not clear enough. If you expect to roll out an application, website or connected device to non-native speakers, be sure to test using native or fluent speakers. Translations are not always accurate; even if they are accurate they might not make sense colloquially. These are errors your users will notice instantly (and likely be very turned off by), but a non-native tester will likely miss. If you plan to roll out the product in multiple languages, use testers to provide extra edits and insight on the translations. Even an hour of usability testing will likely not cover every single possible action on your application. Start by testing those areas of the application that will be used the most or are the most important to your customers. Remember, you can run as many usability tests as you d like (with crowdsourced testing you can even run multiple tests at once with different groups), so don t try to cram everything into one test, it will muddy the waters and result in a less successful usability test. www.applause.com Page 6 of 9
KNOW WHAT KIND OF FEEDBACK YOU WANT While you can typically turn around a usability test quickly with crowdsourcing, you ll get better results if you take time to understand what you re looking for and create a good task plan to get those results. Decide if you want yes/ no answers or more insightful feedback from open-ended questions. Consider which aspects of usability you want covered in a particular test and write the task plan accordingly. When creating a task plan, be thorough, but be careful not to influence the way testers will interact with or feel about the application. Let the kind of feedback you want to receive influence how large the test is. Inviting too many participants to an open-question survey will give you a flood of information to process. If you re looking for input from a lot of people, stick with yes or no questions. If you want more detailed feedback, keep the testing group smaller but give them the opportunity to really explain what they re experiencing and how they feel. If you don t know what type of feedback would be most useful, are concerned about pinpointing actionable data in the test results or are unsure about how best to write a comprehensive task plan, choose a crowdsourcing company that also gives you access to a professional Usability Expert. DON T FORSAKE EXPERTS Just because you re using crowdsourcing doesn t mean you have to sacrifice the knowledge of an expert. A UX expert is a user experience professional; someone who defines the usability testing tasks and questions. They can also help select the right cross section of testers. Once all participants have completed their tasks and answered all questions, the UX expert analyzes the data and writes a results report. The expert also investigates the interface themself for any usability issues based on best practices and industry standards. If your company doesn t have a resident UX professional, the expert s background and experience in user experience and writing of tasks and questions can be extremely beneficial. Even companies that do have an in-house expert or team can benefit from the help of an outside expert who can look at the interface in an unbiased way. Usability testing is far more subjective than functional or security testing. Rather than something working or not working, usability testing is about a user s understanding and feelings about an application. Results based on longform, written feedback can be hard to process, understand and evaluate. Even more difficult, there s not always a clear cut fix to the problem. A UX expert can help separate the fringe opinions from the feedback that really matters and help you understand how to address any issues that come up. If you want the help of a UX expert, be sure you choose a crowdsourced company that offers this service a more self-service company will leave you to do everything yourself. DON T RUSH IT Many people think about running a quick test cycle before releasing or launching their application, website or connected device. Successful companies recognize that usability is a long game that needs continued attention. After all, a user isn t a new user forever. www.applause.com Page 7 of 9
Crowdsourced usability testing gives you the chance to run longer tests without taking time away from internal resources. Longer tests are helpful because they allow you to see how new users change the way they interact with an application over time. There could be features or aspects of an application that only recurring or long-time users will need. Running a longer test will help you accurately gauge the usability of these more advanced features and see how easily users figure them out. Usability test participants aren t sitting in a lab anymore, they re working on their own time in their own environments there s no rush! With the right crowdsourced company you can also have the same users come back to test future versions or updates which will give you insight into how your real customers might feel about the upcoming changes. Crowdsourced usability testing gives you the opportunity to run as many UX tests simulating as many situations over as long a period as you want. Your real users will be interacting with the application for more than two weeks your usability testing shouldn t stop there either. Systems and software should work for us, not make us feel confused, stupid or frustrated. JONATHAN KOHL, SOFTWARE TESTING CONSULTANT CONCLUSION You only get one chance to make a first impression and a good first impression of your product is crucial to keeping customers engaged and happy. If users can t figure out your application, website or connected device quickly, they ll leave quickly and likely won t be back. According to New Relic, 1 in 4 mobile apps are abandoned after the initial use. When asked how to best keep users engaged, Robert Gary, Vice President and General Manager of Mobile Care at Nuance, said, The app must simply be easy to use and avoid frustrating customers. It all comes down to usability. Take advantage of crowdsourced usability testing and work out the kinks (with feedback from a group of people who mimic your real user base) before your application hits the hands of any actual users. Then continue testing to be sure your product will continue to delight them. In the end, an application, website or connected device has to be usable by your users. It doesn t matter if you, your developers and your mom find it usable, if the people in your target market are confused, don t like it or find it frustrating or useless your product is a failure. Crowdsourced usability testing is uniquely positioned to help you avoid that. www.applause.com Page 8 of 9
ABOUT APPLAUSE Applause empowers leading brands to deliver flawless digital experiences for their customers on any device, in every location. Applause market insights, user feedback and digital testing solutions enable businesses to delight customers, drive topline revenue and innovate faster. Applause ensures digital quality for websites, mobile apps, IoT products and in-store interactions in a way no other approach can - via our technology platform and managed global community of over 300,000 professional and ondemand testers. Only real people on real devices in real locations can provide the real issues and feedback that brands need to deliver great digital customer experiences. You can t hire, outsource or automate the increasingly converged digital-physical experience that defines real customer interaction, but you can replicate it with the crowdsourced approach provided by Applause. Thousands of companies of every size including Google, FOX, Nike, BMW, PayPal and Slack rely on Applause to dramatically decrease the time and cost of delivering great digital experiences for their customers. Learn more at www.applause.com AMERICAS INQUIRIES EUROPE INQUIRIES ISRAEL INQUIRIES Applause U.S. HQ 100 Pennsylvania Ave - Suite 500 Framingham, MA 01701 1.844.300.2777 Applause Europe HQ Kopenicker Str. 154 10997 Berlin, Germany +49 30 57700400 11 Galgaley Haplada 1st Entrance - 2nd Floor Herzliya, Israel +972.74.701.4240 0XX-DOCUMENT TITLE-WP-0617 www.applause.com Page 9 of 9