COMPANY CASE STUDY Skyscanner INDUSTRY Online Travel Booking TESTING TYPES Functional APP TYPES Mobile Apps CHALLENGES Uncovering and fixing bugs before they reach customers Introducing localized features into each target market Managing disjointedness that comes with third-party vendors C ustomer Skyscanner is a travel search site that has become a key part of the trip-planning process for 50 million people around the globe each month. Since it was founded in 2003, the startup has steadily grown from a small site used to compare European airfares into a global metasearch engine that enables travelers to browse hotels, rental cars, and both domestic and international flights with a single app. In 2005, it was recognized in The Guardian for finding the lowest cost flights and for beating much bigger operators such as Expedia and Travelocity. Six years later, it joined those same counterparts on Top 10 Travel Sites lists across several other major news publications. Skyscanner has enjoyed seven consecutive years of double-digit growth, and earlier in the year welcomed five new partners, who between them invested $192 million to further accelerate the travel site s growth across the world. RESULTS CEO and co-founder Gareth Williams reflected on the company s motivation, With over 80% of visitors to Skyscanner coming from outside our home market, a focus on delivering to the different needs of travelers wherever they are in the world will remain central to everything we do. Tested in 84 different countries and across more than 1,200 unique device configurations C hallenges Broadened test scope to include a wider variety of defects Accelerating growth and continuously improving the product creates the challenge of ensuring QA keeps up with the fast-paced engineering environment at Skyscanner. Uncovered opportunities to improve efficiency within QA process and bring builds to market even faster Tamás Chrenóczy-Nagy, a Senior Software Engineer for Skyscanner, explains: When we added new features to a build and released it in international markets, we were finding that despite teams focusing on bug-free, high quality code, on occasion bugs were 1
evading our engineers and escaping into production. We needed a safety line, so if we missed something, it would still be caught and fixed before reaching the people using our apps. Additionally, a company with a product that is used across the globe needs to make sure that its digital experience is catering to each of the individual markets that it serves. Our main goal is to localize the content that we have, continued Chrenóczy-Nagy. We are trying to introduce features that are market-specific and optimize our content within those markets. Without a testing presence in each of these specific markets, this type of optimization would be a tremendous challenge. Additionally, working with third-party vendors usually comes with its own set of complications. We didn t want to have to log in to a vendor s web platform to access test results, explained Csaba Szabó, another member of Skyscanner s Software Engineering team. Because that comes with the complicated process of managing extra accounts for all of our team members and teaching them how to use the platform. WE NEEDED A SAFETY LINE, SO IF WE MISSED SOMETHING, IT WOULD STILL BE CAUGHT AND FIXED BEFORE REACHING THE PEOPLE USING OUR APPS. Tamás Chrenóczy-Nagy Senior Software Engineer, Skyscanner Solution Given these considerations, Applause emerged as the optimal solution to meet all of Skyscanner s requirements. Applause met our needs very well, said Chrenóczy-Nagy. It stood far above the other vendors we considered in its ability to test across all of the countries we needed while also improving our efficiency at the same time. After partnering with Applause in 2013, Skyscanner s first order of business was to get test cycles up and running in all of the different countries it needed to as fast as possible. Within days of signing its contract, Skyscanner had a test cycle for its ios app set up with testers located across Spain, Poland, Turkey, Vietnam, Ukraine, Bulgaria, China, Indonesia, and other countries across the globe. From that point forward, Applause has fit seamlessly into Skyscanner s testing cadence. After a day of fixing, updating, and adding features to a build, Skyscanner passes the build to its Applause team for a nightly exploratory test cycle along with a list of focus areas. By the next morning, Skyscanner has a new list of issues and defects to address throughout the day before passing it back to Applause. By testing with Applause, Skyscanner is able to test across more than 1,200 different ios and Android configurations on an ongoing basis. Thanks to the flexibility of Applause, Skyscanner s fears of disjointedness with a vendor have never been realized. We get our issues pushed directly into our own bug tracking system so it 2
is as if our Applause team is really just an extension of our Skyscanner team, explained Szabó. If Applause is an extension of Skyscanner s team, the Applause Project Manager Steve Greenhill could be considered a QA Manager for Skyscanner. Greenhill approves, exports, and triages all bugs reported by the Applause team, and then assigns them to different Skyscanner developers based on issue type. Steve is using the same tools we are using, he is exporting bugs directly into our JIRA system, and he is even included into our Slack, which is our main line of internal communication, explained Chrenóczy-Nagy. If we have questions, we don t need to send emails to the Applause support team, we just message him on Slack and he responds immediately. The total integration with Applause is great. Results Since its first test cycle with Applause, Skyscanner has expanded its QA capabilities to test across 84 different countries on over 1,200 unique Android and ios device configurations. WE GET OUR ISSUES PUSHED DIRECTLY INTO OUR OWN BUG TRACKING SYSTEM SO IT IS AS IF OUR APPLAUSE TEAM IS REALLY JUST AN EXTENSION OF OUR SKYSCANNER TEAM. Csaba Szabó Software Engineer, Skyscanner This expanded coverage has also come with a streamlined approach to testing that has enabled Skyscanner to broaden the scope of its test cycles from Only Critical Issues to All Issues. The process we have today looks completely different than our approach two years ago before we started working with Applause, and those improvements are in part due to valuable consultation with the Applause team said Szabó. When it comes to the quality of the issues Applause is reporting, Skyscanner has described the bugs reported by Applause as very relevant and impactful. Many of the issues Applause reports to us end up approved and fixed, said Chrenóczy-Nagy. So it is clear that Applause is providing valuable feedback to us that is enabling us to bring a higher quality product to market. However, the biggest impact for Skyscanner is the value Applause brings with it. When we were looking to scale our QA, working with Applause made sense based on the cost and coverage it provided, explained Chrenóczy-Nagy. As Skyscanner continues to simplify the process of online travel booking, it will continue to rely on Applause to help ensure that the digital experiences it is delivering will be consistent and flawless in the hands of its users, regardless of where they are in the world. 3
G lobal distribution of testers for S kyscanner : WE COULDN T FIND ANOTHER VENDOR THAT ENABLED US TO TEST ACROSS ALL OF THE COUNTRIES WE NEEDED WHILE ALSO IMPROVING OUR EFFICIENCY AT THE SAME TIME. Tamás Chrenóczy-Nagy 1 25 50 200+ Skyscanner tests across 84 different countries through the Applause community of professional testers. 44
Americas Inquiries Applause U.S. HQ 100 Pennsylvania Ave Suite 500 Framingham, MA 01701 1-844-300-2777 Europe Inquiries Applause Europe HQ Kopenicker Str. 154 10997 Berlin, Germany +49 30 57700400 Israel Inquiries 11 Galgaley Haplada 1st Entrance 2nd Floor Herzliya, Israel +972.74.701.4240 About Applause Applause empowers companies of all sizes to deliver great digital experiences (DX) across web, mobile and IoT as well as brick-and- mortar spanning every customer touchpoint. Applause delivers unmatched in-the-wild testing, user feedback and research solutions by utilizing its DX platform to manage communities around the world. This provides brands with the real-world insights they need to achieve omni-channel success across demographics, locations, devices and operating systems that match their user base. Thousands of companies including Google, FOX, Best Buy, BMW, PayPal and Runkeeper rely on Applause to ensure great digital experiences for their customers. Learn more at www.applause.com. 5