VISUAL TEST AUTOMATION Challenges and Tips. Onur Başkırt

Transcription

1 VISUAL TEST AUTOMATION Challenges and Tips Onur Başkırt

2 Who am I? I am Onur /in/onurbaskirt/ Software Manager at Co-owner Formerly: Teknosa, Ericsson, Huawei, STMicroelectronics, BAU & UCSC

3 Online Recruitment Market Leader in TURKEY since 1999 #1 App in Blue Collar Recruitment Market

4 Software Testing Blog Automation Performance Security DevOps

5 Outline What is Visual Test Automation? Why Visual Testing? Why Visual Test Automation? Challenges Visual Test Automation Tools Takeaways and Tips Q&A

6 What is Visual Test Automation? Verifies the GUI is displayed as expected to the end users. Focuses on visual contents: Colors Shapes Sizes Layout UI Test vs Visual Test

7 Why should We do Visual Testing? UI MUST APPEAR CORRECTLY COMPANY BRAND CORE PRODUCTS and SERVICES MOBILE APP RELEASE PROCESS IS SLOW CSS animations WordPress Template Widgets Sliders etc. Mobile App Generation Engine.ipa.apk

8 Why should We Automate Visual Testing? Variety of Test Environments Browsers Mobile Devices Resolutions etc. Time to Market Pressure Release cycles getting shorter. Manual Visual Testing is Cumbersome Human eyes error-prone and getting blind.

9 What are the Challenges? Anti-aliasing

10 What are the Challenges? Dynamic Content (CSS Animations, GIFs)

11 What are the Challenges? Auto-Kerning Offsets

12 BRIEF OVERVIEW of Well-Known VISUAL AUTOMATION TOOLS

13 There are many tools exists in the market both open-source and commercial. PhantomCSS WebdriverCSS Gemini Wraith FBSnapshotTestCase Needle Rspec Page Regression Pix-Diff Selenium Visual Diff Vizregress VisualCeption Specter Fighting Layout Bugs Dpdxt SiteEffect Shoov BaskstopJS CSSCritic Grunt Photobox VIFF GreenOnion Kobold CSS Visual TestAsd Snap And Compare Grunt-Vigo Hardy Browsershots Screenbeacon VisualReview Screener.io Huxley

14 APPLITOOLS EYES

15 What is Applitools Eyes? Automated Visual Testing Tool for Web sites and Mobile apps Finds Visual Differences with Several Match Levels It has Eyes SDK for Selenium & Appium It has Cloud-Based Clean and Informative Dashboard

16 How Does it Work? At 1 st run take screenshots and save them as baseline images on Cloud At 2 nd run and more, take the same screenshots as actual images and compare them with the baseline images. Applitools Notifies and Reports the visual differences

17 Match Levels and Their Meanings? EXACT Pixel by Pixel Comparison STRICT Mimics the human eyes. Ignores trivial mismatches (antialiasing, small pixel movements, etc.) CONTENT Similar with STRICT plus it ignores color changes. LAYOUT Compares the layouts. Checks the elements locations.

18 4 Main Steps of Applitools Eyes SDK Create and Configure an Eyes Object Eyes eyes = new Eyes(); Start a Test eyes.open(driver, APP_NAME, TEST_NAME); Add UI validation Check Points eyes.checkwindow("validation Window Name"); Close the Test eyes.close();

19 Mobile Visual Testing Example Example Test Result First RUN (It saves BASELINES)

20 Mobile Visual Testing Example Example Test Result PASSED RESULTS

21 Mobile Visual Testing Example Example Test Result FAILED RESULTS

22 Applitools Dasboard

23 GALEN FRAMEWORK

24 What is Galen Framework? Automated LAYOUT and RESPONSIVE testing framework. Runs with Selenium Grid (SauceLabs & BrowserStack) Supports Parallel Testing JAVA & Selenium Webdriver & TestNG/Junit

25 Galen Example with JAVA & Webdriver & TestNG Logo: 31 pixel below from the header Header: 0px to the right, left, top Logo should displayed correctly. (image comparison) Navigation Links: Horizontally aligned.

26 Galen Example with JAVA & Webdriver & TestNG Write a Galen Specifications (.gspec) tagging for desktop web elements Declare Element s location 5 Image Comparison loops

27 Galen Example with JAVA & Webdriver & TestNG Run the Test and Check The Results Galen Test Report

28 Galen Example with JAVA & Webdriver & TestNG Homepage Layout Report

29 Galen Example with JAVA & Webdriver & TestNG LAYOUT ERRORS

30 Galen Example with JAVA & Webdriver & TestNG HEATMAP

31 Galen Example with JAVA & Webdriver & TestNG HIGHLISTS IMAGE COMPARISON ERRORS

32 Visual Test Automation by Using ImageMagick & Selenium & AShot

33 What is ImageMagick? Open-source software suite that manipulates images in a variety of formats (over 200) including PNG, JPEG, GIF, etc. Edits, Converts, Resizes, Compares, Rotates, Adjusts Colors etc. For JAVA it has Im4java java interface.

34 What is AShot? Yandex's screenshot utility which takes, crops, prettifies and compares screenshots. ScreenShot for Entire Page ScreenShot for Web Element

35 Example Test Scenario (Basic CSS Animation Check) Open Unhide text area of Uzman Photo Hover on the UZMAN weblement on main page. Wait 2 seconds to finish CSS animation Take screenshot of an element and save as Baseline HOVER for the 1st Run. For 2nd and more runs COMPARE baseline image with actual image. Put all differences in "Differences folder" BASELINE

36 Example Test Scenario (Basic CSS Animation Check) Test Method s Code JavascriptExecutor Hide Dynamic Content AShot Take Screenshot ImageMagick Compare Images

37 First Run Test Screenshot folder is created by using Test Name

38 Second Run

39 Change The Baseline Test Folder

40 Differences Folder Test Folder Test Failed! Third Run! Arithmetic Error Number

41 TakeAways Visual Testing =! UI Testing ( e.g. Selenium Based Functional Test Automation) Be aware of visual testing challenges and select the right tool. Take screenshots for each environment & browser pair and do comparison for them. If test team do not have enough technical expertise, then invest for trainings. Enhance your software development pipeline with several test automation types: Automated Test Data Generation (Java Faker, Jfairy, faker, Bogus, faker-cs etc. ) API Automation (PostMan, Rest-Assured) UI Automation (Selenium, Capybara, Cucumber, UFT, etc.) Visual Test Automation (Many tools available ) Mobile Automation (Appium, XCUnit, Espresso) Automated Security and Performance Testing (ZAP, Jmeter, Gatling, etc.) Try and use CI & CD & DevOps Practices and Cloud Services (Jenkins, Octopus, Docker, AWS, etc.)

42 THANKS