End-to-End Web Testing: Current Practice and Future Technologies

Transcription

1 End-to-End Web Testing: Current Practice and Future Technologies Paolo Tonella Fondazione Bruno Kessler Trento, Italy Filippo Ricca University of Genova Genova, Italy

2 Background E2E automated Web testing Selenium IDE and WebDriver Page Object Pattern Empirical studies Agenda PO adoption study C&R vs. Programmable study DOM-based vs. Visual locators study Producing robust test code Robula+ prototype Producing cohesive and loosely coupled test code Apogen prototype

3 Testing of Web Apps Web apps are key assets of our society - Business, health care, public administration, - Two billion Internet users worldwide [McKinsey 2011] Correctness is crucial - One bug, 440 millions of dollars lost in less than one hour [Knight Capital Group 2012] NO BUGS! End to End Automated testing - Manual testing is expensive and not effective => automation required! - Low level testing is too complex and not enough => E2E testing!

4 Automated End-to-End Testing Web app functionalities are tested as a real tester does - by feeding them input and examining the output - using the GUI => Black-box

5 Web App under test Test case example

6 Test case example Web App under test Test case (can be executed manually) 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil

7 Test case example Web App under test Test case (can be executed manually) 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil

8 Test case example Web App under test Test case (can be executed manually) 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil

9 Test case example Web App under test Test case (can be executed manually) 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil

10 Test case example Web App under test Test case (can be executed manually) 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil

11 Main Goal of our Research Understanding, on real cases how to reduce the effort of developing and maintaining web test suites Starting point: Analysing E2E web testing approaches Three factors: How web test code is developed (the approach itself) Adoption (or not) of the Page Object Design Pattern How web test code localize the web elements Evaluation conducted by means of empirical studies analysing or building industrial test suites developing and maintaining test suites for open-source web apps

12 E2E Testing Approaches Capture/Replay Recording the actions performed by the tester using a specific tool e.g., Selenium IDE Re-executing them automatically Test Script Test Script Programmable Test code is a software artefact created: Using standard programming languages and IDE e.g., Java + Eclipse Resorting to specific testing frameworks e.g., Selenium WebDriver

13 Web App under test Selenium IDE example Test case 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil Executable test case (Test script) Selenium IDE Test code 1 open type name=q ERC clickandwait id=id_fzi Target/Locator asserttext path=//*[@id='res1']/h3/a ERC: European Research Council Value

14 Selenium WebDriver example Web App under test 1. Open 2. Insert ERC in input field 3. Click submit button 4. Check first result = ERC:EuropeanResearchCouncil Executable test case (Test code) Test case Selenium WebDriver public void searchtest(){ driver.get(" WebElement element = driver.findelement(by.name("q")); Locator element.sendkeys("erc"); element.submit(); WebElement res = driver.findelement(by.xpath("//*[@id='res1']/h3/a")); assertequals(res.gettext(),"erc: European Research Council"); } Assertion

15 Page Object Pattern Strong coupling and low cohesion problem Test code merges two different concerns public void searchtest(){ driver.get(" WebElement element = driver.findelement(by.name("q")); element.sendkeys("erc"); element.submit();... } Test logic + implementation details PO serves as interface of a Web page: it abstracts the functionalities offered by a Web page in methods Implementation details (locators) Indirection layer Test logic

16 Page Object Example Implementation details (locators) Test logic public class LoginPage {... public HomePage login(string UID, String PW) { driver.findelement(by.id("uid") ).sendkeys(uid); driver.findelement(by.id("pw")).sendkeys(pw); driver.findelement(by.id("login")).click(); return new HomePage(driver); } } Test code uses login( ) method offered by LoginPage class

17 Page Objects Adoption Study RQ: Does the adoption of the page object pattern reduce test code maintenance cost? We developed and maintained two equivalent test suites Industrial Web app: elearning Content Management System (LCMS) using Selenium WebDriver with (PatternYes) and without (PatternNO) Page Objects Summary of findings (ICSTW 2013 paper) Adopting POs reduces test code maintenance cost by 3x

18 Why? Change! A change in a Web page can affect only one PO!

19 C&R vs. Programmable study We compared C&R and Programmable approaches: We developed and maintained six test suites using Selenium IDE (C&R) and Selenium WebDriver (Programmable) We measured development and maintenance cost of test code in minutes Summary of findings (WCRE 2013 paper) During development Programmable approach is 1.6x more expensive During maintenance C&R approach is 1.4x more expensive

20 Cumulative cost (development + maintenance) n Cumulative cost of C&R greater after two versions (in the median case)

21 Localisation method Both the approaches (C&R and Programmable) require to locate the web elements to interact with e.g., locate the button to click Two categories of approches: DOM-based locator (XPath) Visual locator //*[contains(text(), Publications )] Based on image recognition algoritms

22 Visual vs. DOM-based study We compared Visual and DOM-based approaches: We developed and maintained six test suites using Sikuli (Visual) and WebDriver (Dom-based) We measured robustness, development and maintenance cost of test code Summary of findings (ICWE 2014 paper) 93% more broken locators in the visual approach During development visual approach is 1.5x more expensive (1.2x for maintenance)

23 DOM based Locators study We studied the reasons of breakage of test code during evolution of six Web apps using Selenium WebDriver (Programmable) Findings indicate that (WCRE 2013) Ids turned out to be the most robust less than 2% of the 459 ID locators were broken, while 60% of the 791 XPath required to be fixed state of the practice XPath locators are very fragile

24 Facts about Locators Ids are the best choice, however Ids don t always exist adding Ids everywhere is impractical or not viable Their uniqueness is not enforced In some cases, they are auto-generated and so unreliable driver.findelement(by.id("id_fzi_1")).click(); So often it is not possible using IDs locators An alternative is using XPath expressions

25 Lesson Learnt: E2E testing not a silver bullet! 1. Fragility problem Minor changes to the Web app can invalidate/break the existing test code Test Script Web page Evolution Test Script Web page invalid!

26 Lesson Learnt: E2E testing not a silver bullet! 1. Fragility problem Minor changes to the Web app can invalidate/break the existing test code Test Script Web page Evolution Test Script Web page invalid! Web page Test Script Test code too coupled with the page structure and so difficult to understand and maintain! 2. Strong coupling and low cohesion problem Test code merges two different concerns Test logic/steps Implementation details Maurizio Leotta, Diego Clerissi, Filippo Ricca, Paolo Tonella. Approaches and Tools for Automated End-to-End Web Testing. Advances in Computers, Volume 101, pp , Editor: Atif Memon. Elsevier, 2016

27 CHALLENGE 1 Producing Robust Test Code

28 Robust XPath locators Empirical data show that robust XPath locators contain few names, predicates and levels Level of an XPath: number of / or // symbols Short XPath locators are more robust! M. Leotta, A.Stocco, F. Ricca, P. Tonella. Reducing web testcases aging by means of robust XPath locators. Proc. of Int. Symposium on Software Reliability Engineering Workshop, ISSREW, 2014.

29 Shorter is better: an example DELETE: HTML Page Select the director name: L1 /html/body/div[2]/table/td[2]/text() breaks if the left div is deleted Fragile XPATH L2 content ]/*/td[2]/text() L3 80% ]/td[2]/text() do not break if the left div is deleted Reduced dependency on entire HTML structure

30 ROBULA+ (ROBUst Locator Algorithm) GOAL: Generating automatically XPath locators that are ideally as robust as the best DOM-locators that developers can produce DOM D input ROBULA+ output Abs XPath of element e Robust Locator for e in D Maurizio Leotta, Andrea Stocco, Filippo Ricca, Paolo Tonella. Robula+: an algorithm for generating robust XPath locators for web testing. Journal of Software: Evolution and Process 28(3): (2016)

31 ROBULA+ (ROBUst Locator Algorithm) Start from the most general XPath expression ( //* ) apply a number of transformations until a locator is produced: Transformations: transfconvertstar //*/td //tr/td transfaddid //td //td[@id= name ] transfaddtext transfaddattribute //tr/td //tr[@name= data ]/td transfaddattributeset transfaddposition //tr/td //tr[2]/td transfaddlevel //tr/td //*/tr/td Several heuristics prioritize the generation of the results - Transformation ordering - transfaddlevel is the last - Attributes Prioritization - robust attrs inserted first - e.g., id, name - Black List - fragile attributes excluded - e.g., src, href, tabindex

32 Example - ROBULA+ ROBULA+( /html/p[1], D)

33 Example - ROBULA+ ROBULA+( /html/p[1], D) //*

34 Example - ROBULA+ ROBULA+( /html/p[1], D) //p First iteration //* //*[text()= X ] //*[@class= a ] //*[1] //*/* No one is a unique locator so we go for a second iteration

35 Example - ROBULA+ ROBULA+( /html/p[1], D) //* Second iteration //p //*[text()= X ] //*[@class= a ] //*[1] //*/* //p[text()= X ] //p[@class= a ] //p[1] //*/p Returned Xpath (unique locator)

36 ROBULA+ evaluation We evaluated its effectiveness on 1110 web elements

37 ROBULA+ evaluation FirePath absolute FirePath relative id-based Selenium IDE Montoto algorithm Robula+ reduces locator maintenance by 3x Indeed, Robula+ has: Smallest levels Lowest use of positions

38 CHALLENGE 2 Producing cohesive and loosely coupled Test Code

39 Generating POs automatically APOGEN tool public class SearchPO { private final WebDriver private WebElement private WebElement result; public void search (String s){ query.sendkeys(s); query.submit(); } public string getfirstresult (){ return result.gettext(); } } public void searchtest(){ } driver.get(" SearchPO SPO = new SearchPO(driver); SPO.search("ERC"); assertequals(spo.getfirstresult(), "ERC: European Research Council"); A. Stocco, M. Leotta, F. Ricca, P. Tonella, APOGEN: automatic page object generator for web testing. Software Quality Journal, pp 1 33

40 APOGEN: the approach (Automatic Page Object GENerator) Web Application FSM FSM [minor modifications] Crawler Clusterer Cluster Visual Editor [clustering not satisfactory] State Object-based Model Page Objects for Web Application Code Generator Static Analyser

41 Navigational methods Hyperlinks are extracted to create the PO navigational methods

42 Action methods Forms are extracted to create the action PO methods

43 Getter methods Clustering Clustering similar concrete pages (states) and computing the diff Portion of the page (potentially) relevant for assertions DIFF Owner Information PO String getname() {...} String getaddress() {...} String getcity() {...} String gettelephone() {...}

44 Code Generator Code Gen. Static analyzer + Clusterer

45 Experimental results % % %

46 Questions? APOGEN