How We Learned To Love Test Automation for Localization AUTOMATED TESTS

Size: px

Start display at page:

Download "How We Learned To Love Test Automation for Localization AUTOMATED TESTS"

Ernest Bradford
5 years ago
Views:

1 How We Learned To Love Test Automation for Localization AUTOMATED TESTS

2 Executive Summary In general, localization testing is manual as text needs to be read, layout needs to be checked, and localization needs to be verified all of which require a person to manually perform. But, automation is possible in localization testing too, enabling the localization testers to be more efficient. We wrote this white paper to help you understand the importance of an integrated test automation framework, and to elaborate on the idea how we learned to love test automation, and how you can learn to love it too. This white paper will guide you through the steps we took with our clients to build UI Automation, to make localization testing more effective and efficient. This transition from manual to automation testing reduced our client s test cases by 60%, increased test coverage by 23%, and saved time by 97.5%. 2

3 What We Hated About Test Automation There was lot of technical debt in the code There was a need for lot of manual efforts to run automation Silk lacked a way to run tests in parallel Silk lacked data driven capabilities Silk lacked multi-browser support and backward compatibility with IE Testing Pyramid Developers couldn t reproduce bugs found in Silk, since Silk environment was complex Developers did not want to use point & click tools, they wanted to code There were 80,000 NRT test cases and a 20-day NRT cycle. Each of our teams took days to run NRT, with localization tests taking the longest time We Also Wanted to Go Agile Testing Matrix GUI tests are brittle, or was there a way to make it less brittle through design? The goal was to have manual testing team focus on new features using exploratory testing, and automate as much of the tests as possible 3 Focus on Functional, GUI, and Content tests. Business facing + critique product quadrant must be manual, or are there ways to make manual linguistic testing more effective?

4 Agile Testing Pyramid Before Agile transformation, most of the testing logic was written in silk and all business logic, integration logic, and GUI logic were tested in silk at the GUI layer. By default, GUI tests are brittle, since any change in the UI will break the tests. SeLion enforces the best practice of separating the UI elements and the UI logic, which will increase the robustness of the GUI tests. Testing Tools Manual Session Based Testing Brittle Tests Selenium SeLion Automated GUI Tests FitNess REST API Cucumber Automated API Tests Automated Integration Tests Automated Component Tests xunit TestNG Automated Unit Tests 4 Solid

5 Symbio Automation Framework Jenkins Product Build o Web o Mobile Deploy TEP (Test Execution Platform) Framework Metrics &Reports Screenshots Defect reporting Repository Test Scripts Dashboard JIRA LQA Review 5

6 Symbio Automation Dashboard Web Mobile Desktop 6

7 Screenshot library Start -> Search -> Shop -> Sort -> Select -> Buy -> Configure -> Cart All test flow pages are captured as screenshots 7

8 Review and Bug reporting Bug review window gives clear view of details with Markup highlight feature Bugs can be reported from Dashboard to JIRA, filling most of the data automatically 8

9 UI Test Automation with SeLion Selenuim is a great open source solution, however: All tests were flow-based vs page-based Selenium only supports web based testing Selenium didn t support multi devices Selenium didn t support multi lingual This took us part of the way there. Luckily, Selenium gave us more options: Much easier, cheaper to add new core functionality The large Selenium community provided us with drivers to drive native apps on ios and Android We could easily add whatever we were missing, then roll it out to the community for their use, in turn allowing them to add on, and improve it Solution: Extend Selenium Into SeLion Multi-device Multi-platform Multi-lingual Browser Web zh j a ko th hi ar ms p t ru vi Phone Tablet TV Mobile t r pl cs f i fa sv da n l i d ro Desktop Desktop 9

10 How Did We Do It? We added what was missing, added more features, and released it into the open source community. This gave us: A separation of concerns: YAML to handle page objects + multilingual support Test cases focused on test logic YAML provided the ability to fix page changes in one place instead of having to fix in every script Very readable test scripts Testing grid, Jenkins + Selenium grid to run tests Testing environment component to provide data driven tests Added Android + ios drivers that plugged into Selenium grid, providing the multiplatform support Example Test Case Script YAML: Page Objects + Locale baseclass: com.paypal.selion.testcomponents.basicpageimpl pagetitle: US: Official Apple Store defaultlocale: US elements: officialapplestoreapplestorelink: locators: US: //div[@id= page ]/div[2]/div/div[1]/h1/a/span ZH: //div[@id= page ]/div[2]/div/div[1]/h1/a/span searchsubmitbutton: locators: US: //button[@class= gh-search-submit gh-search-magnify ] ZH: //button[@class= gh-search-submit gh-search-magnify ] sitesearchquerytextfield: locators: US: //input[@id= site-search-query ] ZH: //input[@id= site-searchquery ] pagevalidators: [ officialapplestoreapplestorelink ] 10

11 SeLion Design UI in multiple languages - Web pages - Mobile screens - Desktop screens Test Code Yaml Yaml Yaml Template - Page Objects - Multi-lingual support Preprocessor Page Object Repository Test NG Suite Tests 11 Tests - Page flows - End-2-End test

selenium-web driver bindings for node.js.

12 UI Test Automation with Nemo.JS Why Nemo? Nemo combines mocha.js, the confit configuration engine, plugin architecture, and flexible parallel options to the selenium-web driver bindings for node.js. With all the goodies that SeLion has, Nemo is smaller and more Node.js friendly Allows automation testing to be part of your Node.js app s development life cycle. For Node.js application 12

We will be able to: A separation of concerns: Data Provider to handle test case level and locale default data to feed the test.

13 How Did We Do It? Nemo.JS is a open source NPM package. We will create functions that fit test projects, and released it as Nemo plugins. We will be able to: A separation of concerns: Data Provider to handle test case level and locale default data to feed the test. Test suites focused ontest logic Page Object to handle page locaters and page level common steps. Less maintenance effort. Testing on cloud Test run in parallel in data driven. Example Test Case Script JSON: page locators, localized data 13

14 Nemo Work Flow A P P L I C A T I O N N e m o Test Suites Page Object Data Provider Locale Data NPM Case Data JENKINS Validation Layer(Screenshots + Check Points) Reporting Layer Test Reports Screenshots Stack Traces /Slack Notifications 14

15 UI Automation with Cucumber/Appium Write Test Scripts in Human Language Write test scripts in human language Cucumber uses Gherkin to define test cases, which is a language designed to be human readable, and easily accessible to non-technical people Dev and QA can help to write automation scripts easily Multi-languages, Multi-platforms, Multi-devices support as well as SeLion platform Native/Hybrid/Webview Apps supported Browser Web Phone Tablet TV Mobile Desktop Desktop 15

16 How Did We Do It? We created functions and test steps that fitted test projects, and implemented to framework. Supported data driven tests Test steps were defined and centralized Human readable scripts Easy onboarding Supported Native/Hybrid/Webvie w Apps, and Supported multiple OS/system on devices farm Example Test Case Script Steps pre-defined 16

17 Cucumber/Appium Framework Design Test cases Test cases Test cases Multiple languages cases - Native case - WebView test - Hybrid Support Libraries JENKINS Appium/UIAutomator2/Frank Device Farm/Cloud Predefined Steps APP/Browser Tests Run tests on multiple devices/os 17

18 How We Learned To Love Test Automation? Results Running tests in parallel greatly decreased total run time Full automated NRT cycle down to 2 days, shortening the cycle For localization tests, 20 days went down to 4 hours 60% reduction of test cases, one team went from 3000 to 800 cases Product and testing teams were able to develop test cases Every release, 90% of the tests were run using automation New tests per release focused on new features, and were SBTM based Automation cost was lower than manual testing cost, yet the coverage was much higher. The test cases were cumulative over time Time Saving by 97.5% Test Cases Reduction by 60% Cost Coverage 20 days 4 hours $ $ $ $ $ $ $ 80 % 98 % 18

19 Symbio Can Help You to Implement UI Automation How can we help you? Symbio UI Automation solution emerged out of the need for a better test automation framework Symbio UI Automation solution can help you to drastically improve your processes Symbio UI Automation solution will scale your testing and QA, and you will be able to test more and test better in a shorter period of time Symbio UI Automation solution will help your testing teams to be more efficient Symbio UI Automation solution will increase the test coverage, and will minimize test cases SeLion will also drastically lower the cost of testing Manual Testing Multilingual Multiplatform Multidevice Automated Testing 19 en fr de it es fi SYMBIO Automation

20 Contact Us Symbio is the global leader in innovation, co-creation and seamless augmentation. We provide insight and strategy, design and research, productization, software development, quality assurance, globalization, R&D operations, systems management and support operations for over 400 customers worldwide. Visit us online at or contact us directly at

We re redefining Software Quality

We re redefining Software Quality Continuous Testing Web Services Agile Testing Mobile Device Farm Test Lifecycle Management Performance Test Quality Assurance Mobile Device Management Test Life Cycle