Unit testing (OHJ-306x)

Transcription

1 Unit testing (OHJ-306x) Tools, techniques, tips and tricks Mika Maunumaa Researcher Tampere University of Technology

2 Introduction What is unit testing? Quick and short intro Terminology review The assignment (Firefox 3.6 alpha 1) Test target Usage scenario Setting up the test environment Tools for unit testing Reading and comprehending the code Things about C-language

3 Unit testing in big picture Testing levels Acceptance testing Carefully selected tests verifying that the system works as customer requires In real environment System testing Tests verifying that the system works are specified In laboratory or real environment Integration testing Depending on abstraction level Tests interaction between modules Tests interaction between subsystems In simulated or laboratory environment Unit testing Testing external behavior of a unit or small cluster of units

4 Unit testing Testing for individual building blocks of a system A unit can be a module, a class or a function Logically bound set Unit Under Test (UUT) MUT, CUT, FUT etc. In object-oriented testing the definition of unit can be a bit vague When object cluster is big enough to be a unit When the cluster becomes too large Integration testing A unit depends on other units Those that uses and drives drivers Imitates the real usage of the UUT Those that are used by UUT stubs Imitates the functionality of the real code Test Suite Group of tests Group of test suites Tests for same part of a unit Tests using same technique Test Case Test sequence with purpose A set of test data, test sequence and expected result Test Case has three stages: Setup and initialization Test sequence Cleanup and reset

5 Unit Test Case execution phases 1a) Test setup Constructs the test setting Prepares stubs for particular test case Correct or incorrect answer Hard to find error situation Disk Full, Memory Full, etc. 1b) Test initialization Prepares UUT Correct initial state Connects UUT to stubs, if necessary Reserves resources DB connection, TCP/IP connection, dynamic memory etc. 2 ) Test sequence exercises the UUT Driver executes usage scenario, verifies the result and prepares the verdict If top-down approach is used, the indirect driver can be used Actual driver exercises indirect driver (some system class) that exercises the actual unit under test 3a) Test cleanup free reserved resources 3b) Test reset unwinds the test setting Depending on test environment and framework you are using, each phase may or may not overlap with next phase

6 Making unit test cases Problem: where to acquire knowledge about UUT Specification for external behavior Black-box testing Mozilla provides no documentation, make assumption about the behavior Reading the code White-box testing Requires code reading and comprehending skills No documenting comments in code code is a document Social engineering Common coding rules and how coders normally avoids to using them Testing techniques Boundary values, NULL-pointers, domain violations Test data tools to extract (good) test data

7 Making unit test cases In general, when you don t have interface specification or the specification is rubbish, you as a tester have to: Use your domain knowledge and practical experience to make educated guess about the interface specification Look for the code for hints that might give you the clue how it should work Count on coverage metric tools and create tests that cover 100% of the code Write tests that verifies if the code works as it is coded (???) Each method should have at least one TestSuite of their own Basic functionality tests for all methods can be in the same TestSuite Each test case should be as small as possible, but not smaller Each test sequence should test one thing and one thing only If there are two or more sequences places in one test case, the failure in one will break the sequence perfect and/or importance test sequence are ignored

8 Making unit test cases The only successful test is the one that finds bugs No bugs found no bugs in the system poor testing A bug is a social creature, if there is one, there can be more Testing more you can only rise your confidence about systems quality However, test which verifies if the unit is behaving as it should are important Enough test cases? Test until you drop Deadline McCabe s cyclomatic complexity metric estimate for lower boundary of the number of test cases Multiconditional coverage -> boolean statements every truth value combination will be tested If (a>1 && b<5) { (a=1,b=5), (a=2, b=5), (a=1, b=4), (a=2,b=4)} Short circuit mechanism

9 Making unit test cases Multiconditional coverage requirement 100 % required, but less can be reached It is only a mean to measure ENOUGH, not the end If you are pursuing only for maximum MC coverage you will forget to test Test contains ALWAYS four parts 1. Prepare (initialize UUT and other data) 2. Execute (prepare method parameters and execute method/function) 3. Examine return values and compare them to expected values 4. Clean up How do you call sequence that does only 1, 2, and 4? A function call Is not a TEST

10 Control flow graph of ParseURL [speclen < 0] speclen < 0 speclen = strlen(spec) IF 3 rivillä 126 [stop!= 0] [colon!= 0] [colon > stop] [stop == 0] [colon == 0] [colon <= stop] IF 4 [return NS_OK] r [speclen!= 0] FOR 2 FOR 1 [len!= 0] [*p > '\0'] [*p <= ' '] [*p <= ' '] [p!= spec] IF 3 [*p!= '\0'] [true *p!= '\0'] [colon == 0] [true colon == 0] [slash == 0]] [*p == ':'] [*p == ':'] [*p == '/'] [*p == '?'] [*p == '#'] [colon!= 0] [slash!= 0] then haara r [!net_isvalidscheme] [net_isvalidscheme] [colon < slash] [slash!= 0] [*(colon+1) == ':'] [*(colon+1)!= ':'] [*p > ' '] [colon == 0] [colon!= 0] [colon >= slash] [slash == 0] [p == spec] else haara r [authoritylen == 0] [pathlen == 0] [authoritylen!= 0] [pathlen!= 0] [*p == ';'] NS_ERROR_MALFORMED_URI [authoritylen!= 0] [*p == '@'] [*p == '['] [stop!= 0] [authoritylen == 0] [pathlen!= 0] [pathlen == 0]

11 Unit test assignment Test testing target is Firefox browser s URL-parser Firefox version 3.6 alpha 1 Parser is line and character based Parses line from left to right UUT is class called nsstdurlparser In file nsurlparsers.cpp (.h) Methods that must be tested (McCabe) ParseURL (32) ParseAfterScheme ( 7) ParseFilePath (12) ParseFileName (6)

12 Unit Test Configuration package UUT [ Test Environment ] nsurlparsers nsiurlparser nsiurlparser nsbaseurlparser +ParseURL() +ParseServerInfo() +ParsePath() #ParseAfterScheme()... +net_isvalidscheme( s : char"*", l : int ) nscautostring nsnoauthurlparser nsauthurlparser +ParseAuthority() +ParseUserInfo()... -str : char"*" -len : int <<constructor>>+nscautostring( s : char"*", l : int ) +ToInteger( err : PRInt32"*" ) : PRInt32 UUT nsstdurlparser +ParseAfterScheme() Stub interfaces

13 UUT usage scenario spec = speclen = strlen(spec) NS_IMETHOD ParseURL(const char *spec, PRInt32 speclen, PRUint32 *schemepos, PRInt32 *schemelen, PRUint32 *authoritypos, PRInt32 *authoritylen, PRUint32 *pathpos, PRInt32 *pathlen); http :// /~testaus spec2 = (spec + authoritypos + 1) // speclen = authlen + pathlen - 1 void ParseAfterScheme(const char *spec2, PRInt32 speclen, PRUint32 *authpos, PRInt32 *authlen, PRUint32 *pathpos, PRInt32 *pathlen) // /~testaus serverinfo = (spec + authoritypos) // serverinfolen = authoritylen NS_IMETHOD ParseServerInfo(const char *serverinfo, PRInt32 serverinfolen, PRUint32 *hostnamepos, PRInt32 *hostnamelen, PRInt32 *port) // :8080 port == 8080 path = (spec + pathpos) pathlen2 = pathlen /~testaus NS_IMETHOD ParsePath(const char *path, PRInt32 pathlen2, PRUint32 *filepathpos, PRInt32 *filepathlen, PRUint32 *parampos, PRInt32 *paramlen, PRUint32 *querypos, PRInt32 *querylen, PRUint32 *refpos, PRInt32 *reflen) /~testaus Rest output paramaters are 0 and -1 (*Pos and *Len)

14 Tools Setting up the unit test environment for assigment Copy assignment files to your working directory mkdir ~/testaus cd ~/testaus /share/tmp/testaus/scripts/setup_unittest cd unittest CTC Coverage measurement tool from Testwell Instruments the source code with measuring code Executed before compilation gcc file.cpp ctc i m gcc file.cpp (instrument with multiconditional cov.) Collects data to mon.dat file Cumulatively, so delete this file before your final test and delivery Requires some environment variables For every login you must execute. /share/testwell/bin/testwell_environment Reading results ctcpost MON.dat less ctc2html for html output Consult their man pages

15 CppUnit - unit test framework Unit test declaration Unit test implementation #include <cppunit/extensions/helpermacros.h> class TestCAutoString : publiccppunit::testfixture { CPPUNIT_TEST_SUITE( TestCAutoString ); CPPUNIT_TEST( testnormalstring ); CPPUNIT_TEST_SUITE_END(); public: void setup(); void teardown(); void testnormalstring(); void testnullstring(); }; #include "testcautostring.h" #include "nsiurlparser.h" CPPUNIT_TEST_SUITE_REGISTRATION( TestCAutoString ); void TestCAutoString::setUp() { // before each test function } void TestCAutoString::tearDown() { // after each test function } void TestCAutoString::testNormalString() { // test steps here }

16 Make and Makefile (MTP 15.5) Build management Compilation rules Compilation targets Assignment contains a Makefile and there is only one line that must be modified... # Do not modify anything else but the list of object # files required by the "unittest" target: #!!! ADD HERE CPPUNIT TESTCASES unittest: unit.o stubs.o testrunner.o testsomething.o $(CTCCC) -o $@ $^ $(LINKFLAGS) stubtest: stubs.o testrunner.o testcautostring.o testutilityfunctions.o unit.o... You should not need to modify anything else

17 Things that they did not taught in school C-string A pointer to memory location Always ends with null character \0 Vector of 8 bit integer values Common translation to ASCII characters Pointer shows the head p == 0x100 The length of the string p is 5 strlen(p) = 5 length of char vector itself is 6 How to access data *p == 72 == H *(p+5) == 0 p[3] == 108 == l C-String cannot be copied as an object char * s=p copies only value 0x100, not the string itself char *s = strdup(p) // free s after use std::string ss(p) Common security risks Do not use C-String functions that does rely on the presence of trailing null character Use strnxxx-functions that takes the maximum number of characters as a parameter p Hello\0 0x100 0x101 0x102 0x103 0x104 0x H e l l o \0

18 Odd things while implementing stubs Do not return stack variable int * func() { int bar = 5; return &bar; } Do not store stack variable to class member Do not store vector pointer parameter to class member Content may have been deleted void someclass::func(const char * p, int length) { strclassmember = p; strclassmember = strndup( p, length ); } Do not rely on atoi-function atoi( Hello World ) == 0 Sometimes it is enough to implement simple stub Test data can order UUT to always call the stub with certain data Implement stub that statically returns what is wanted and error otherwise c-string functions often introduces security risks strdup vs. strndup char * strndup( psrc, copylength )

19 Thank you Questions!