T-76.5613 Software Testing and Quality Assurance Automated testing in Agile SW development Seppo Sahi SoberIT seppo.sahi@soberit.hut.fi 2.10.2006
Introduction Agile methods have strong emphasis on practices that support good quality in the constructive work Pair programming, war room, on-site customer etc. Much of the testing activities are actually constructive work Test-driven development Less testing with destructive attide Agile methods have a very strong focus on quality issues Testing in agile environment Automated unit testing Automated acceptance testing Exploratory testing 2
No manual tests Chapter about manual tests in Testing Extreme Programming (Crispin & House 2003) No manual tests. Quick and minimal effort regression testing Keep up a good pace in software development Enables safe refactoring Suitable testing tools not always available Organization has to develop own tools 3
Agile acceptance testing with FIT Acceptance testing = A level of testing conducted from the viewpoint of the customer, used to establish the criteria for acceptance of a system.
Requirements and acceptance testing Agile requirements definition Specifying requirement in great detail is delayed until the details are really needed Agile acceptance testing In XP Testers, customer and developers design acceptance test together main responsibility in customer Acceptance tests are designed before implementation All acceptance tests are gradually automated Developers use tests as executable specifications Acceptance testing is not a phase in the end it is a tool to carry out a project 5
Roles Turning customer requirements into an automated test is a team effort Customer - knows what is needed Design and write tests together with testers and programmers Tester - knows how to test software and where software systems usually break Write more advanced tests (e.g. negative tests, use testing techniques) Programmers are not interested Customers don t have the required skills Consider testing viewpoint early Programmer connects tests and application 6
What is FIT and FitNesse? FIT (Framework for Integrated Test) Framework for Acceptance testing FitNesse A Wiki based UI to create, edit and run FIT tests Non-programmers write and run tests Business people, customer etc. A little help from testers and programmers is needed Standard and convenient way to write tests, log results, group tests into suites GUI and command-line based execution tools Available for many programming languages 7
Creating and modifying acceptance tests is easy Use Word, Excel or HTML editor 8
How acceptance tests plug-in to application (1/2) Simple glue code a.k.a Fixture code is needed Framework offers a variety of ready fixtures from which programmers can inherit their own ones Handles parsing of test data etc. Separation of test data and test code Test data Fixture code Application code Presentation Fixture test fixtures Application under test 9
How acceptance tests plug-in to application (2/2) Presentation test fixtures Business rule test fixtures Fixture code Application code UI Presentation API Client-side presentation, validation, calcualtion Business rule API Business rules 10
Demo Writing and organizing tests Running tests Writing fixture code 11
References FIT homepage - http://fit.c2.com/ Download framework and tools Example code Articles about agile acceptance testing Crispin, Lisa. 2005, Using Customer Tests to Drive Development, Methods & Tools, Summer 2005. http://www.methodsandtools.com/pdf/mt200502.pdf Shore, James. 2005, A Vision for Fit. http://www.jamesshore.com/blog/a-vision-for-fit.html Marick, Brian. 2004, Driving Projects with Examples: a Handbook for Agile Teams. http://www.exampler.com/book/ Reppert, Tracy. 2004, Don t Just Brake Software. Make Software., Better Software, July/August 2004. www.industriallogic.com/papers/storytest.pdf 12
Unit testing with JUnit Seppo Sahi SoberIT seppo.sahi@soberit.hut.fi
Unit testing A unit is the smallest possible testable software component Various interpretations exists Procedure / function Class / object / method Small-sized component Goal is to ensure that software units are functioning as intended E.g. a function returns correct value when exercised with valid parameters E.g. function behaves correctly when given an invalid input Automated unit test == a piece of code that exercises another piece of code 14
Unit testing - a vital testing level Writing software is difficult and every programmer makes mistakes There will be defects in all software you write The cost of a defect is lowest when it is found in unit testing Defects that slip to upper levels (integration, system or acceptance testing) More difficult to spot Time consuming to debug Trivial bugs on unit level can be hard to resolve on system level Uses organizational resources Writing, prioritizing and managing test and defect reports Bug found during the use in production Loss of money, loss of... Unit testing lays the foundation for good quality software Higher levels of testing are needed to ensure overall quality of software Unit testing lays foundation for effective testing on higher levels Bad unit level quality slows down integration and system testing 15
How do you know that your code works correctly? Do you just trust your luck? Do you use printf outputs to see what happens in your code? Do you use debugger to find out how your code really works? Do you ask your friend/colleague to check through your code? Do you create small (temporary) widget to access the piece of code directly runtime? Do you use some ugly, cumbersome, time consuming practice that leaves trash into your code and is thrown up to the user s face when you least expect it Getting here is not possible, something is really messed up in this code!!! Fatal error 9981: Assertion failed : av_d_dx > 0 16
Unit testing frameworks Convenient frameworks for writing unit tests as part of the programming task In the same environment with the same language Tools of xunit family are the most famous ones junit CppUnit NUnit The unit testing frameworks provide a convenient way of writing and executing unit tests Standard and convenient way to write tests, log results, group tests into suites GUI and command-line based execution tools Integrated to IDEs Convenient report generators Xml, html 17
Demo Framework features IDE integration 18
Naming conventions for junit tests Naming conventions tell what class and method is under test 1. import junit.framework.testcase; 2. 3. public class TriangleTest extends TestCase 4. { 5.... 6. 7. /** 8. * Tests that {@link Triangle#classify()} can classify an 9. * impossible triangle. 10. */ 11. public void testclassify_withimpossibletriangle() { 12. Triangle impossibletriangle = new Triangle(8, 3, 4); 13. 14. String classification = impossibletriangle.classify(); 15. String expectedclassification = "impossible"; 16. 17. assertequals("one side longer than two others together" + 18. "should be impossible.", 19. expectedclassification, 20. classification); 21. } 22. 23.... 24. } 19
Naming conventions for junit tests (2/2) CODE: Triangle.java TEST FOR THE CODE: TriangleTest.java classify() testclassify() isisosceles()...() testisisosceles() testisisosceles_withequilaterialtriangle() testisisosceles_withnonisoscelestriangle() Test method name Begins with test......followed by the name of the method it tests......and side conditions of the test test...() test...() 20
Basic test flow Create test data (12) Perform operation and fetch result (14) Specify expected result of the operation (15) Check that result matches the expected (17-20) Clean up 1. import junit.framework.testcase; 2. 3. public class TriangleTest extends TestCase 4. { 5.... 6. 7. /** 8. * Tests that {@link Triangle#classify()} can classify an 9. * impossible triangle. 10. */ 11. public void testclassify_withimpossibletriangle() { 12. Triangle impossibletriangle = new Triangle(8, 3, 4); 13. 14. String classification = impossibletriangle.classify(); 15. String expectedclassification = "impossible"; 16. 17. assertequals("one side longer than two others together" + 18. "should be impossible.", 19. expectedclassification, 20. classification); 21. } 22. 23.... 24. } 21
Simple Test class 1. import junit.framework.testcase; 2. 3. public class VectorTest extends TestCase { 4. private Vector testvector = null; 5. 6. protected void setup() { 7. testvector = new Vector(); 8. } 9. 10. protected void teardown() { 11. testvector = null; 12. } 13. 14. public void testisempty() { 15. asserttrue("vector is not empty", testvector.isempty()); 16. } 17. 18. public void testaddelement() { 19. testvector.add(new Object()); 20. assertequals("wrong size!", 1, testvector.size()); 21. } 22. 23. public static void main(string[] args){ 24. junit.textui.testrunner.run(vectortest.class); 25. } 26. } 22
Assert method summary 1/2 assertequals(string msg, Object expected, Object actual) Compares two values for equality. The test passes if the values are equal. Comparison methods for all primitive data types, too (int, float, long, byte, char..) asserttrue(string msg, boolean b) Evaluates a boolean expression. The test passes if the expression is true. assertfalse(string msg, boolean b) Evaluates a boolean expression.the test passes if the expression is false. assertnull(string msg, Object a) Compares an object reference to null. The test passes if the reference is null assertnotnull(string msg, Object o) Compares an object reference to null. The test passes if the reference is not null. 23
Assert method summary 2/2 assertsame(string msg, Object a, Object b) Compares two object references using the == operator. The test passes if both refer to the same object. assertnotsame(string msg, Object a, Object b) Compares two object references using the == operator. The test passes if both refer to different objects. fail(string msg) Causes the current test to fail All asserts take a string as the first parameter Provides documentation Shows up in the JUnit TestRunner interface Good messages can tell you what is wrong without starting up the debugger Check possible assertxxx() methods from JUnit JavaDocs! 24
Unit testing best practices (1/3) Write automated unit tests rather than use printf or debugger Automated unit tests retain value over time Tomorrow, after five years,... No need for human interaction Write once run with minimal effort Test code should contain as little logic as possible Introducing more logic will introduce more errors Don t use loops, branches etc. Unit tests should be independent Don t assume a specific execution order Don t assume specific execution time Tests have no side effects 25
Unit testing best practices (2/3) Write tests like they were small increments Each test tests very small and targeted subject Tests build on each other - utilize existing tests to create new ones Keep test methods short Kent Beck: Long tests indicate the likelihood of a design problem Refactor your code so that it can be tested in smaller parts JUnit is designed to stop on the first failing assert on purpose! Use method naming conventions Test name should describe functionality tested and side conditions E.g. testadduserwithalreadyexistinguser Document your unit tests Use method naming conventions Write JavaDocs for test methods Include a message in assertxxx() 26
Unit testing best practices (3/3) Treat test code like production code Remember readibility and clarity Remember coding conventions Store data in ways that can be managed by the source control system Test data evolves along with test cases If it can't break on its own, it's too simple to break otherwise, write unit tests testing setx() and getx() usually ignored Run all your unit tests as often as possible, ideally every time the code is changed. For larger systems, you may just run specific test suites that are relevant to the code you're working on Keep all tests running Tests have to be maintained like application code 27
Agile unit testing and test-driven development
Document driven vs. agile unit testing Principle 7. Testing should be carried out by a group that is independent of the development group Course book presents a unit testing viewpoint which appreciates this principle Unit testing should be separate task conducted after the development is done Unit test design is based on written specifications Modern ideas of unit testing represent slightly different viewpoint Unit tests are written by developer Automation is essential part of unit testing Unit tests might even be written before the actual program code This lecture is mostly about agile unit testing Read the course book Ch. 6 and understand the difference The same tools are used by both disciplines 29
Agile unit testing Test-driven development is practiced Writing and running tests is integral part of daily development rhythm Developers QA and programming tool All unit tests are automated Automated tests are run as often as possible Gather all your tests to a test suite and use it for regression testing Integration, nightly, milestone During development; after every change to the code; new code or bugfix 30
Test-driven development Writing test before code to be tested a little test, a little code, a little test, a little code,... Tests are added gradually during implementation not in large lump afterwards Process of writing tests drives low-level design and programming Tests specify what code should do Tests validate that code does what it should Actually, a design and coding practice One of the core practices of Extreme Programming Developers have been applying TDD for several decades 31
TDD episode (1/2) Proceeds step by step 1. Write a test. Write test 2. Design and implement just enough to make the test pass. 3. Repeat. Testing and coding alternate in very small steps Duration of one cycle should be a few minutes Fail Write code Run tests Pass Small steps difficult to make mistake 32
TDD episode (2/2) Episode is over when you can t write a failing test anymore Write test for each requirement of the code Write test for each point that can possibly break One cycle at a time Don t write a bunch of tests at once Refactor if you ever see the chance to make the design simpler Run all tests after finishing episode Make sure you did not brake anything else 33
Demo: CommandLineParameters class Class that represents command line parameters given Format should be /<param> <value>, e.g. /backup true /interval 300 34
Test-Driven Development claimed benefits (1/2) Close feedback loop TDD cycle is very short know if code is working right after you programmed it Task-orientation Encourage programmer to decompose problem into manageable programming tasks Helps to maintain focus Helps to measure progress and scope work Low-level design Programmer is forced to think which classes and methods to create, how they are used, how to name them, what arguments does a method take, what does a method return 35
Test-Driven Development claimed benefits (2/2) Results better code If the test is too hard to write, the code being tested is too complicated Results testable code Programmer can t end up with code that cannot be tested Effect on quality Testing becomes part of the development process and gets done Side effect of TDD is that code gets thoroughly unit tested 36
Try it! The only way to know! Personal experiences Good feeling about the code written General confidence that your code does what you have intended it to do Good feeling when checking your code into version control with all green Tests really get written when they are written beforehand You allways have an up-to-date regression testing suite TDD helps you to keep focus on the current task Program only what is needed to see the green light Promote best practices System.out.println is used for displaying messages for user not for developer Debugger is used for debugging 37
Example of testbase growth in a project that used TDD 4 developers, iteration length 2 weeks 12000 10000 8000 6000 4000 2000 0 Pre-project Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Iteration 6 Maintenance Application Testcode Comments Comments in testcode SQL code 38