Testing. Technion Institute of Technology Author: Assaf Israel. Author: Assaf Israel - Technion

Transcription

1 Testing Technion Institute of Technology Author: Assaf Israel

2 Why test? Programming is incremental by nature We want to verify we haven t broken anything Tests not only examine the code s functionality but also its Ease of use API Boilerplate & setup Error path behavior Reveals coherency issues Good tests forces the production code to be modular (decoupled) Tests are the usage example of our code and are an excellent source for documentation 2

3 Testable Code Good Code Tests are usually the first time outside code will use your production code! Is the production code hard to test? Do you write LOC before you reach the assert line? Is it difficult to assert the desired behavior? Do you find it hard isolating the tested feature? Are you not even sure what you should be asserting? Is the production code hard to use? Is object creation hard and complex? Are methods cryptic with a lot of parameters? How does your production code handle usage errors? Have you been surprised by a method behavior? 3

4 Test types Our focus today Unit Tests Test a single element of the system A single element does not necessarily imply a single class Integration Tests Verify that a few units can work together and are wired up correctly End-to-End Tests Test the complete flow of interactions throughout the system Acceptance / Business logic Tests Answers the question: Did we build what we wanted? 4

5 Test types 5

6 Testing Framework Wish list Better Asserts: Validating results Vs. expectations assert(sqrt(4) == 2) Vs. assertequal(2,sqrt(4)) assert(x!= null) Vs. assertnotnull(x) Clear indication of Success / Failure Meaningful message upon failure Ease of use Should be easy to add new Tests Aggregate tests together into Test Cases / Tests Suites Continue with other tests even if a test fails / throws an exception No duplication of initialization code of tested objects Side effects should not influence other tests 6

7 JUnit A unit testing framework for Java A Test is a Method A Test Case is a class which includes tests methods Aggregates tests for a single tested unit A Test Suite is a collection of Test Cases / Test Suites (or both) Used to create a hierarchy of tests 7

8 JUnit 4.0 A step up from JUnit 3.0 New Assertions methods in org.junit.assert.* No need to extend from junit.framework.testcase No need to prefix test methods with test Test methods are runs once before first test Setting up the stubs objects / DB runs once after the last test Closing DB method runs before every test Preparing data Could be used for advanced method runs after every test Resetting stubs / cleaning data 8

9 JUnit test case - Running all methods annotated with A Test Case Class load Static C C tor C tor C tor @After* 9

10 JUnit life-cycle - Example Expensive and global resources can be initialized and destroyed before/after all tests public class DbTests { static DatabaseConnection public static void login() { database public void testsomething() public void testsomethingelse() public static void logout() { database.logout(); 10

11 JUnit life-cycle - Example Local resources should be initialized and cleaned before/after every test public class DbTests { File public void createoutputfile() { output = new public void testsomething() public void testsomethingelse() public void deleteoutputfile() { output.delete(); 11

12 JUnit Example Assuming we want to test a new Stack interface implementation import org.junit.assert; import org.junit.test; public class StackTestCase { Name is testing public void peekreturnsthelastpusheditem() { Stack<String> s = new ArrayListStack<>(); s.push("first-item"); s.push("last-item"); Assert.assertEquals("LAST-ITEM", s.peek()); Expectation Initialization Result org.junit.comparisonfailure: expected:<[last]-item> but was:<[first]-item> Smart Asserts: assertnotequal, asserttrue, assertsame, assertnotnull, assertarrayequal, etc. 12

13 Ideal Test A recipe for a good public void testshouldnameexplicitlystatesexpectation() { (1) Initialize collaborators (2) Initialize object under test (3) Exercise behavior (4) Assert outputs Ideally: (1) will be very short (2) will be simply a new MyClass() (3) will be a single method invocation (4) will be a single assert Of course this ideal cannot always be achieved But if you can t, you should always ask yourselves why? 13

14 Test names It should be easy pin-pointing the problem from the test without even looking at its code Consider the following public void additemtest() { Product p = new Product(); p.setavailable(false); ShoppingCart cart = new ShoppingCart(); cart.additem(p); assertequals(0, p.numitems()); How about additemwithoutofstockproduct? Describes the method and the parameter Bad name. Specifies the name of the invoked method but not describes the desired behavior Still not a good name does not describes the desired behavior 14

15 Test names It should be trivial understanding what is the expected behavior from the name alone We do not need to mention the name of the invoked public void donotaddoutofstockitem() { Product p = new Product(); p.setavailable(false); ShoppingCart cart = new ShoppingCart(); cart.additem(p); assertequals(0, p.numitems()); Good rule of thumb: the test name should include an explicit expectation Use verbs such as should, does, returns, throws, etc. 15

16 JUnit Exceptions A test that throws an exception will fail. So how do make sure a code throws an exception when it should? We would like to verify 3 things: The right exception has been thrown The exception contains the right message The right line has thrown the exception 16

17 JUnit Exceptions The hard way Use the try/catch public void badencodeargument() { Encoder encoder = new Encoder( UTF-8 ); try { encoder.encode(null); fail( Exception should have been thrown ); catch (NullPointerException e) { assertequals( Text should not be null,e.getmessage()); Verifies our 3 wishes Easy to forget the fail resulting in a test that never fails Quite long and verbose Test code is just as important as our regular code! We want to minimize code duplication and boilerplate. 17 Based on

18 JUnit Exceptions The easy way (JUnit 4.0) expected value = IllegalArgumentException.class) public void badencodeargument() { Encoder encoder = new Encoder( UTF-8 ); encoder.encode(null); Can t check the exception message Can t verify which line has thrown the exception Very short Can be good enough in many cases, but we can do better 18 Based on

19 JUnit Exceptions The smart way (JUnit 4.7) annotation and public ExpectedException thrown = public void badencodeargument() { Encoder encoder = new Encoder( UTF-8 ); thrown.expect(nullpointerexception.class); thrown.expectmessage( Text should not be null ); encoder.encode(null); Verifies our 3 wishes Still quite short JUnit Rules offers a few additional candies 19 Based on

20 JUnit - Timeout We want to verify that our code finishes in the allotted public void testtimeout() { final boolean[] done = new boolean[] { false ; new Timer().schedule(new TimerTask() public void run() { foo(); done[0] = true; Test will fail after 10 milliseconds., 0); Thread.sleep(10); if (done[0] == false) fail( Failed to run in the allotted time"); This is criminally verbose Surely, there is a simpler way 20

21 JUnit - Timeout In JUnit 4.0 Use the timeout annotation = 10) public void testtimeout() { // Tested code Test will fail after 10 milliseconds. Global timeouts (JUnit public MethodRule globaltimeout = new public void test1() { // Tested public void test2() { // Tested code 21

22 JUnit Ignoring tests You can ignore tests using annotation Test will not run, but will still be visible in the test reports Commented out tests will usually be forgotten about Won t be deleted and possibly will not be fixed Ignored tests are easily visible in the output window Make sure you disable tests temporarily Bad tests should either be deleted or fixed Ask yourself why are you ignoring the test in the first place? 22

23 JUnit - Running Textual runner Used mainly in command line environments (Scripts, Maven, Ant, CI) org.junit.runner.junitcore.runclasses(testcase1.class,...); IDE Support To run several TestCase1.class, TestCase2.class, subpackage.alltests.class, ) public class AllTests { 23

24 JUnit Source code Organization Option 1: Test class in same folder as subject A single hierarchy of folders Locality Option 2: Keep tests in the mirrored directory structure of source location A duplicated hierarchy of folders Easy to find all tests Easy to separate source code from its tests Easier bytecode separation 24

25 Testing Leveraging OOP Tests are classes too! Can use inheritance to our advantage Use Abstract Test classes against abstract classes / interfaces Classes Tests Class A ATest B C BTest CTest 25

26 Testing Leveraging OOP public abstract class Stack<T> { public void push(t element); public int size(); public class ArrayStack<T> extends Stack<T> { // implementation code public class LinkedListStackStack<T> extends Stack<T> { // implementation code 26

27 Testing Leveraging OOP public abstract class StackTest { private final Stack<Object> stack; protected StackTest(Stack<Object> stack) { this.stack = public void sizeisoneafteronepush() { stack.push(new Object()); Assert.assertEquals(1, stack.size()); 27

28 Testing Leveraging OOP Extend abstract TC to initialize the concrete test subjects public class ArrayStackTest extends StackTest { public ArrayStackTest() { super(new ArrayStack<Object>()); // specific ArrayStack tests public class LinkedListStackTest extends StackTest { public LinkedListStackTest() { super(new LinkedListStack<Object>()); // specific LinkedListStackTest tests 28

29 Testing Order (or lack thereof) Do not assume order between tests There s no order guaranties in TestCases There is one in TestSuites Avoid side effects in tests public class StackTest { private static Stack<Object> s = new public void pushfirstandchecksize() { s.push(new Object()); public void pushsecondandchecksize() { s.push(new Object()); assertequals(2,s.size()); 29

30 Testing Resource Management Do not load data from hard-coded locations on a file public void setup () { InputStream inp = FileInputStream("C:\\TestData\\dataSet1.dat"); Use Class.getResource() or public void setup () { InputStream inp = getclass().getresourceasstream("dataset1.dat"); Will search the file relatively to the Test Case.class file Maven will automatically create a src & test resource folders 30

31 Unit Testing: What to Test It is not practical to test everything: State space > 2 #arguments+#members Not always practical Loops in the code State space is infinite / Halting Problem E2E and Acceptance tests will cover the state space quite well Contracts (later in the course) Cover as much of the code as possible, but be smart about it Extensiveness of the test should be proportional to the complexity of the impl. of the tested method If m2() calls m1(), no need to duplicate m1() tests for m2() But, for each variation in the flow (usually an if()) you should have a corresponding test that exercises it Always test public methods Should you test package-private and protected methods as well? Private methods usage is important only in the context public methods If you find yourself wanting to test a private method, you should rethink your design, (e.g., extract the private method to a helper class) 31

32 Unit Testing: Quality Cover code usage Check common usage errors / edge cases: nulls empty strings zeros empty lists single-item lists uninitialized objects bad configurations Exception handling Code Coverage Function, Statement, Branch, Decision coverage Monitor using external Code Coverage tools (e.g. EclEmma) 32

33 Code Coverage - EclEmma 33

34 Code Coverage Code coverage is correlated with good testing, but does not equate it You can achieve very high coverage with bad tests (e.g., testing without asserts) You can write great tests with low coverage 100% coverage may look pretty, but is pointless You should always suspect tests that were written solely for achieving high coverage You can use code coverage to Spartanize your code e.g., branches that are never taken may be dead code 34

35 Testing Do s & Don ts Always place shortest, simplest tests first in a TestCase or in a TestSuite You should always make sure the simplest cases work before you carry on to the complicated ones Prefer lots of short tests instead of a few long ones Every test should test a single behavior Speed up discovery-to-fix cycle Testing code should be simpler than the subject code Complicated test usually implies complicated production code usage Tests are the code usage examples Do not put if() statements in your tests. Loops are quite rare. 35

36 Testing Do s & Don ts Always make sure you start Green before you start coding. Unless working in TDD Run tests as often as possible The earlier you detect the bug the sooner you solve it and the easier it becomes Regression testing Make sure new additions don t break existing code When you work on a bug, write a test case first Your work is done when the test case succeeds If you write the test after the bug is fixed, you cannot know if the test succeeded in reproducing the bug 36

37 Testing Do s & Don ts When you change the subject class, add tests Testing is meant to improve the quality of the code Do not write bad code to pass tests!! Although possible in TDD Do not write bad tests to pass tests!! (Make sure a test can fail) Do not simply change a private method into a public one so you can test it It will probably be wiser to extract the method into a new class, and then make it public/package Hard to test? Re-think the design! 37