Test Automation to the Limit

Size: px

Start display at page:

Download "Test Automation to the Limit"

Janice Carter
5 years ago
Views:

1 Test Automation to the Limit Arie van Deursen Delft University of Technology Test Automation Day, 23 June,

2 Outline 1. Background Joint work with Ali Mesbah (UBC), Danny Roest (TU Delft) Michaela Greiler (TU Delft) 2. Testing Web Apps to the Limit 3. Testing Plug-ins to the Limit 4. Discussion 2

3 The TU Delft Software Engineering Research Group Group size: ~30 fte, including ~20 full time researchers (PhD students, postdocs). Teaching: Programming, projects ~20 MSc projects p/y in software engineering Research: Software evolution Reverse engineering Software testing Software architecture End-user programming Software services Model-driven engineering 3

4 4

5 Test Automation? What benefits can your technique achieve? Within what limits is your technique successful? What s going wrong if move beyond those limits? What can we do to move the limit? 5

6 Part I: Testing Web Apps to the Limit Asynchronous Javascript And XML 6

An Ajax Fault Model Stateful client; Asynchronous communication; Delta updates; Untyped JavaScript; Client-side DOM manipulation; Event handling; Timing; Back/forward,

7 An Ajax Fault Model Stateful client; Asynchronous communication; Delta updates; Untyped JavaScript; Client-side DOM manipulation; Event handling; Timing; Back/forward, undo/redo Browser dependence (IE, FF, Chrome, Focus on client side GUI logic. Different fault models / test strategies for server side / database / business etc. logic 7

8 Testing Web Applications Traditional: Response Request No client side logic Capture & Playback: Selenium Manual effort Can we push these limits? 8

9 9

10 State: DOM Tree Edge: Event that is fired and causes a state transition 10

11 Steps Involved Finding (new) clickables Triggering events Comparing DOM trees Recurse Backtrack Input data generation 11

12 Implementation: Crawljax Open source: Java, maven, JUnit, Apache libraries Relies on webdriver to access the browser IE, Firefox, Chrome Plugin-based architecture Contributions from NL, Canada, Japan, Used at Microsoft, Google, Fujitsu, ebay, 12

13 13

14 14

15 Scalability Indicators 15

16 Crawling for Testing Purposes? Crawling = Testing? Oracle? Browser eats all issues Leverage derived model 16

17 Invariants as Oracles Generic, on the DOM-tree state validity, error messages in DOM, accessibility, link discoverability, security... Generic, between successive DOM states Consistent back-button Application-specific (design) invariants: Constraints on specific element and attribute relations in particular DOM states 17

18 Example Invariants Pointing to Faults 18

19 Past Executions as Oracles Application to Regression Testing 1. Infer a model of a trusted version 2. Create test suite with the model 3. Run test suite on newer version 4. Check all the links and compare the states ICST

20 Finding & Visualizing Relevant Differences 20

21 Cross-Browser Testing Step 2: Establish pairwise equivalence of the generated models Web Application + Crawljax?? Step 1: Crawl web app. under different environments and capture behavioral model under each environment Execution environment (OS + Browser) Models of crawled behavior 21

22 - AdSense

23 Example: Accessibility testing 23

24 Web Application Testing Benefits? Fully automatic Built-in oracles Several types of testing Beyond the limits? Scalability Unknown territory Limits? Nr of states Benefits in the trenches? Pushing the limit? Concurrent crawling State abstraction Joint case studies 24

25 Part II: Testing Plug-in Architectures Create series of tailored products by combining, configuring, & extending plug-ins 25

26 26

27 One Product = Many Plug-ins Set of Plug-ins = Many Products Fault model: Interacting plug-ins, Plug-in configurations Plug-in versions Plug-in extensions Resource usage Test practices: Are these tested? Why? How? Why not? 27

28 Unit testing is popular Unit testing is where you find the most bugs We don t have UI tests, because we can cover that with unit tests P18 P3 At least 70% of our test effort is spent on unit testing. P16 28

29 Cross plug-in testing is optional We do cross plug-in testing bug-driven. P18 P19 We have no automated tests for cross-plugin testing, but we do manual testing. 29

30 Version testing is minimal A lot of people put version ranges in their bundle dependencies, and they say they can run with 3.3 up to version 4.0 of the platform. P13 But I m willing to bet that 99% of the people do not test that their stuff works. 30

31 Automated or Manual? 31

32 Organizational Problems Experienced 32

33 Technical Impediments Experienced 33

34 Discussion What benefits can your technique achieve? Within what limits is your technique successful? What s going wrong if move beyond those limits? What can we do to move the limit? 34

35 Test Automation to the Limit 35

36 More information

웹소프트웨어의신뢰성. Instructor: Gregg Rothermel Institution: 한국과학기술원 Dictated: 김윤정, 장보윤, 이유진, 이해솔, 이정연

웹소프트웨어의신뢰성. Instructor: Gregg Rothermel Institution: 한국과학기술원 Dictated: 김윤정, 장보윤, 이유진, 이해솔, 이정연 웹소프트웨어의신뢰성 Instructor: Gregg Rothermel Institution: 한국과학기술원 Dictated: 김윤정, 장보윤, 이유진, 이해솔, 이정연 [0:00] Hello everyone My name is Kyu-chul Today I m going to talk about this paper, IESE 09, name is "Invariant-based