Time - Experience Report. By Thanou Thirakul

Transcription

1 Large Scale Testing In Agile Time - Experience Report

2 Large Scale Testing In Agile Time - Experience Report What we re going to talk about: 1. Background on the application 2. Background on the build process and its problems 3. Four points that address the problem and a fifth point that includes my own observations about integration testing.

3 Large Scale Testing In Agile Time - Experience Report Here s the Story The Application is a system for filling drug prescriptions. It s being created by approx. 15 developers, which we ll call the Rx Team It has been in development for 6 years It has over 1 million lines of code! It uses Eclipse Standard Widget Toolkit as a user interface

4 Sample SWT UI

5 Large Scale Testing In Agile Time - Experience Report Here s the building and testing ti part The Rx team used CruiseControl to build and run tests

6 Sample View each build Build statistics Build History

7 Large Scale Testing In Agile Time - Experuence Report Here s the Building and Testing part The RX team used CruiseControl to build and run tests It ran over 8,000 unit tests (8% of the build time) It ran over 1,200 client integration tests on six machines (75% of the build time) It took two hours to run

8 It had many successful production releases!

9 Not too So, bad. what s So what s the the problem?

10 These are the problems A failed build A successful build The time a build starts Build dates

11 So, these are the problems the Rx team was facing. The team lost faith in the build process It had a big complex build script that no one wanted to touch There were too many false build failures The integration tests were taking too long

12 Large Scale Testing In Agile Time - Experience Report How did the Rx team fix the problems? 1. The team asked for help. They went to a special proup in Intelliware that deals with process issues (the Process Group ) 2. The Process Group came up with a plan. 3. This plan resulted in the four points I m Im going to talk about. Each point highlights a problem and a solution.

13 Point #1 Problem: The team lost faith in the build process

14 Large Scale Testing In Agile Time - Experience Report Problem: The Rx team lost trust in the build process so they ignored the build results. Result: They bypassed the build and deployed the code to the server untested. When untested code goes on a server it can create a lot of bugs. Solution: A special team was created. Its only goal was to find ways to solve the build and testing process. New people were brought into this team including me.

15 Point #2 Problem: It had a big, complex build script that no one wanted to touch

16 Large Scale Testing In Agile Time - Experience Report Solution: Separate build activities iti We divided build activities into build artifacts run tests Use a build tool Use Maven to manage artifacts

17 Point #3 Problem: There were too many false build failures

18 False failures caused by client environment failures

19 Large Scale Testing In Agile Time - Experience Report Problem: There were too many false build failures because Testing puts lots of stress on a machine s environment Over time the environment degrades Solution: Use virtual machines for test clients because it has the ability to reset the environment for each test run.

20 Large Scale Testing In Agile Time - Experience Report Solution: Use virtual machines for test clients because it has the ability to reset the environment for each test run. Virtual test clients

21 Point #4 Problem: The integration tests were taking too long

22 Large Scale Testing In Agile Time - Experience Report Solution: Build your own tool if you can t find one Challenge: We needed to build a tool that t could manage and distribute ib t manage 1,200 integration tests on more machines to reduce the run time

23 We built a Test Server!

24 What is a Test Server? It s a central service that t assigns tests t to client machines to run. It manages current and past test t runs.

25 Solution: Build your own tool if you can t find one here s the tool we built. Test Server Client Do you have a test t for me to run? Yes, run Test A Done. Here s the result. Do you have more? Yes, run Test B Done. Here s the result. Do you have more? runs Test A runs Test B No more test. You re done. exits

26 How does it work with many clients? Runs Test G Client 7 Client 6 Assigns Test F Runs Test F Client 5 Runs Test E Assigns Test E Assigns Test F Assigns Test A Assigns Test D Test Server Assigns Test C Assigns Test B Client 4 Runs Test D Client 1 Client 2 Client 3 Runs Test A Runs Test B Runs Test C

27 Solution: Build Your Own Tool If You Can t Find One Looking at the real thing 13 tests running in parallel

28 Solution: Build Your Own Tool If You Can t Find One Results: We achieved the same time using 3 physical boxes (with 13 virtual machines) versus 6 physical boxes from the old build.

29 Point #5 My own observation: Integration testing

30 Large Scale Testing In Agile Time - Experience Report Observation: Integration ti tests t are expensive Find balance Gave the Rx team a solution they can make their process even better. Cost Value

31 Large Scale Testing In Agile Time - Experience Report Summary 1. Create a special team to solve specific problems 2. Separate build activities iti 3. Machine environment stability is important 4. Build your own tool if you can t find one 5. Integration tests are expensive

32 Thank You Thanou Thirakul