Software Testing 101 or Why Should We Bother With Software Testing?

Transcription

1 Software Testing 101 or Why Should We Bother With Software Testing? Crash Course Lecture Marat Akhin Saint-Petersburg State Polytechnical University December 8th, 2009 Marat Akhin (SPbSPU) Software Testing /12/ / 51

2 Contents 1 Prelude 2 Basics 3 Design for testability 4 Coverage 5 Regression testing 6 Resume Marat Akhin (SPbSPU) Software Testing /12/ / 51

3 What we WON'T cover today Heavy formal stu Heavy formal stu Input: T 1, T 2 AST äëÿ ïðîãðàììû äî è ïîñëå ìîäèôèêàöèè Output: M ìíîæåñòâî ñâÿçàííûõ óçëîâ äëÿ T 1 è T 2 1: M 2: Îòìåòèòü âñå óçëû T 1 è T 2 êàê ¾íåñâÿçàííûå 3: for x T 1 do Îáõîäèì T 1 ñíèçó ââåðõ 4: if y T 2 : ismatched(y) match(x, y) = 1 then 5: M M (x, y) 6: Îòìåòèòü x è y êàê ¾ñâÿçàííûå 7: end if 8: end for Marat Akhin (SPbSPU) Software Testing /12/ / 51

4 What we WON'T cover today Details of testing techniques Details of testing techniques Marat Akhin (SPbSPU) Software Testing /12/ / 51

5 What we WON'T cover today Specic testing software Specic testing software Marat Akhin (SPbSPU) Software Testing /12/ / 51

6 What we WILL cover today Software testing 101 Software testing 101 Some basic things about software testing that you should know... Marat Akhin (SPbSPU) Software Testing /12/ / 51

7 What we WILL cover today Software testing 101 Software testing 101 Some basic things about software testing that you should know......if you want to be a good software tester Marat Akhin (SPbSPU) Software Testing /12/ / 51

8 What we WILL cover today Software testing 101 Software testing 101 Some basic things about software testing that you should know......if you want to be a good software tester Should you learn ¾Software testing 101 if you want to be a good software developer? Marat Akhin (SPbSPU) Software Testing /12/ / 51

9 What we WILL cover today Software testing 101 Software testing 101 Some basic things about software testing that you should know......if you want to be a good software tester Should you learn ¾Software testing 101 if you want to be a good software developer? YES Marat Akhin (SPbSPU) Software Testing /12/ / 51

10 What we WILL cover today Software testing 101 Software testing 101 Some basic things about software testing that you should know......if you want to be a good software tester Should you learn ¾Software testing 101 if you want to be a good software developer? YES I hope you'll understand why in about one and a half hours =) Marat Akhin (SPbSPU) Software Testing /12/ / 51

11 What we WILL cover today Why should we bother with software testing? Why should we bother with software testing? Short answer: because I tell you that you should do so! Marat Akhin (SPbSPU) Software Testing /12/ / 51

12 What we WILL cover today Why should we bother with software testing? Why should we bother with software testing? Short answer: because I tell you that you should do so! Long answer: because experience of thousands and thousands of developers all over the world indicate that software testing really helps in the development of quality software. Otherwise why is it used in almost every software development methodology? Marat Akhin (SPbSPU) Software Testing /12/ / 51

13 Prelude What question does testing answer? What question does testing answer? Does this software work? NO, testing cannot answer that Marat Akhin (SPbSPU) Software Testing /12/ / 51

14 Prelude What question does testing answer? What question does testing answer? Does this software work? NO, testing cannot answer that Does this software fail? YES, testing can (and should) answer that Marat Akhin (SPbSPU) Software Testing /12/ / 51

15 Prelude What question does testing answer? What question does testing answer? Does this software work? NO, testing cannot answer that Does this software fail? YES, testing can (and should) answer that Testing = Destruction Marat Akhin (SPbSPU) Software Testing /12/ / 51

16 Prelude What ¾testing is and what isn't What ¾testing is and what isn't It is: Best friend of Verification and Validation What is the dierence between the two? Verification checks whether the software is correct and right Validation checks whether the software complies to a set of requirements In other words: We verify that ¾we did it right We validate that ¾we did the right thing Marat Akhin (SPbSPU) Software Testing /12/ / 51

17 Prelude What ¾testing is and what isn't What ¾testing is and what isn't It isn't: Way to ensure 100% guarantee of any kind The software will never fail with NPE The database will never deadlock Calculations are always correct Time requirements are always fullled Marat Akhin (SPbSPU) Software Testing /12/ / 51

18 Prelude What ¾testing is and what isn't What ¾testing is and what isn't It isn't: Way to ensure 100% guarantee of any kind The software will never fail with NPE The database will never deadlock Calculations are always correct Time requirements are always fullled We ¾can give 100% guarantee but only in the most trivial cases and testing itself is usually not needed when we talk about trivial stu Marat Akhin (SPbSPU) Software Testing /12/ / 51

19 Prelude What ¾testing is and what isn't What ¾testing is and what isn't It isn't: Way to ensure 100% guarantee of any kind The software will never fail with NPE The database will never deadlock Calculations are always correct Time requirements are always fullled We ¾can give 100% guarantee but only in the most trivial cases and testing itself is usually not needed when we talk about trivial stu We'll cover why 100% guarantee is impossible a little bit later Marat Akhin (SPbSPU) Software Testing /12/ / 51

20 Prelude Why testing is so dicult Why testing is so dicult Theorem Software testing is all about PAIN Proof. Software engineering is all about PAIN Software testing Software engineering Marat Akhin (SPbSPU) Software Testing /12/ / 51

21 Prelude Why testing is so dicult Why testing is so dicult Software Method Build-and-x Waterfall Model Extreme Programming Rapid Prototyping Unied Process Pain It just doesn't scale up We can't predict everything at the start Writing good test cases is very dicult Throwing away the prototype is hard Following its many rules is a pain Dan Berry: ¾The Inevitable Pain of Software Development Fred Brooks: ¾No Silver Bullet Marat Akhin (SPbSPU) Software Testing /12/ / 51

22 Prelude Why testing is so dicult Why testing is so dicult Brian Kernighan ¾Debugging is twice as hard as writing the code in the rst place. Therefore, if you write the code as cleverly as possible, you are, by denition, not smart enough to debug it. Marat Akhin (SPbSPU) Software Testing /12/ / 51

23 Prelude Why testing is so dicult Why testing is so dicult Brian Kernighan ¾Debugging is twice as hard as writing the code in the rst place. Therefore, if you write the code as cleverly as possible, you are, by denition, not smart enough to debug it. Massimo Arnoldi (feat. Kent Beck) ¾Unfortunately at least for me (and not only) testing goes against human nature. If you realize the pig in you, you will see that you program without tests. Marat Akhin (SPbSPU) Software Testing /12/ / 51

24 Prelude Why testing is so dicult Why testing is so dicult Brian Kernighan ¾Debugging is twice as hard as writing the code in the rst place. Therefore, if you write the code as cleverly as possible, you are, by denition, not smart enough to debug it. Massimo Arnoldi (feat. Kent Beck) ¾Unfortunately at least for me (and not only) testing goes against human nature. If you realize the pig in you, you will see that you program without tests. We have a very serious problem Now we understand that software testing is really hard. But we have to do it to make quality software. A very simple question what do we do? Marat Akhin (SPbSPU) Software Testing /12/ / 51

25 Prelude What do we do? What do we do? Stick our teeth Learn how to minimize the pain of testing Understand how the testing works Understand how software testing can help software development Understand how software development can help software testing Believe in software testing Marat Akhin (SPbSPU) Software Testing /12/ / 51

26 Basics A very basic scheme of software testing A very basic scheme of software testing Run the software Check the run results hoping to nd errors aka bugs aka faults aka defects aka aws aka failures Let's clear up the denition of ¾software error, shall we? Marat Akhin (SPbSPU) Software Testing /12/ / 51

27 Basics Model of software error Model of software error FAILURE FAULT ERROR Failure observable from the outside incorrect program behaviour Fault corrupted program state caused by an error Error developer's mistake in a program Let's check this model on an example Marat Akhin (SPbSPU) Software Testing /12/ / 51

28 Basics Model of software error (example) Model of software error (example) Find the error in the following Java program 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Marat Akhin (SPbSPU) Software Testing /12/ / 51

29 Basics Model of software error (example) Model of software error (example) Find the error in the following Java program 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Possible overow at line 4 Marat Akhin (SPbSPU) Software Testing /12/ / 51

30 Basics Model of software error (example) Model of software error (example) c = {} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Marat Akhin (SPbSPU) Software Testing /12/ / 51

31 Basics Model of software error (example) Model of software error (example) c = {} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } No fault and no failure program works correctly Marat Akhin (SPbSPU) Software Testing /12/ / 51

32 Basics Model of software error (example) Model of software error (example) c = {1, 2, 3, 5} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Marat Akhin (SPbSPU) Software Testing /12/ / 51

33 Basics Model of software error (example) Model of software error (example) c = {1, 2, 3, 5} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } No fault and no failure program works correctly Marat Akhin (SPbSPU) Software Testing /12/ / 51

34 Basics Model of software error (example) Model of software error (example) c = {1, 2, 3, 5, +MAX_INT, -MAX_INT} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Marat Akhin (SPbSPU) Software Testing /12/ / 51

35 Basics Model of software error (example) Model of software error (example) c = {1, 2, 3, 5, +MAX_INT, -MAX_INT} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Program goes through corrupted state (fault is present) But no failure visible from the outside Marat Akhin (SPbSPU) Software Testing /12/ / 51

36 Basics Model of software error (example) Model of software error (example) c = {1, 2, 3, 5, +MAX_INT} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Marat Akhin (SPbSPU) Software Testing /12/ / 51

37 Basics Model of software error (example) Model of software error (example) c = {1, 2, 3, 5, +MAX_INT} What happens? 1 i n t s u m C o l l e c t i o n ( f i n a l Collection < Integer > c ) { 2 i n t sum = 0; 3 f o r ( i n t i : c ) { 4 sum += i ; 5 } 6 return sum ; 7 } Program goes through corrupted state (fault is present) And it fails with incorrect results Marat Akhin (SPbSPU) Software Testing /12/ / 51

38 Basics How to nd a software error How to nd a software error Model can be represented as: a mental representation of ¾how this software should work requirements from software technical specication dierent test cases set of correct software outputs another (a priori correct) implementation of the same specication Marat Akhin (SPbSPU) Software Testing /12/ / 51

39 Basics How to nd a software error How to nd a software error Testing run must ensure the following 3 properties to actually nd an error: Reachibility test must execute (¾reach ) the software error Corruption error must corrupt the program state to produce software fault Propagation fault must persist through the program run and cause a failure in program behaviour Making these properties hold is a major pain in software testing Marat Akhin (SPbSPU) Software Testing /12/ / 51

40 Basics How to nd a software error How to nd a software error To soothe this pain, we have to develop a program that is: Controllable we can easily make the program go where and how we want it to go Observable we can easily understand what the program is doing at any given time If we don't think about that before we start developing chances are we'll never be able to make the program controllable and observable. We need ¾Design for testability Marat Akhin (SPbSPU) Software Testing /12/ / 51

41 Design for testability Test-driven development Test-driven development Most well known technique of ensuring ¾Design for testability Created by Kent Beck back in 1999 Two simple principles: We write test cases rst Then we make them work by writing code Forces the developer to create and maintain a good software specication Makes the software controllable and observable: Each test case corresponds to a specic control ow trace thus giving us controllability Each test case checks a set of software outputs thus giving us observability Marat Akhin (SPbSPU) Software Testing /12/ / 51

42 Design for testability Controllability Controllability Simulation and stubbing We can easily control our own code but we can't do that with third-party code like OS routines, incomplete code, etc. We have to simulate or stub (S&S) third-party code to control what it is doing S&S may be faster than third-party code S&S can easily produce hard-to-induce error Marat Akhin (SPbSPU) Software Testing /12/ / 51

43 Design for testability Controllability Controllability Simulation and stubbing We can easily control our own code but we can't do that with third-party code like OS routines, incomplete code, etc. We have to simulate or stub (S&S) third-party code to control what it is doing S&S may be faster than third-party code S&S can easily produce hard-to-induce error Downwards scalability Many bugs live in ¾the dark corners of software limits To make them appear we have to reach these limits and it is easier to do so when the limits are nice and small We scale the limits down to reach them during testing Marat Akhin (SPbSPU) Software Testing /12/ / 51

44 Design for testability Observability Observability Assertions Improve observability by propagating some errors into failures And we can observe the error directly not just the results of it Marat Akhin (SPbSPU) Software Testing /12/ / 51

45 Design for testability Observability Observability Assertions Improve observability by propagating some errors into failures And we can observe the error directly not just the results of it Invariant checkers Extension of assertions to do a full-scale program state checking Inspect program state at some points during execution to ensure state correctness Of course we run invariant checkers only during testing they are too expensive to run during actual program run Marat Akhin (SPbSPU) Software Testing /12/ / 51

46 Design for testability Observability Observability Assertions Improve observability by propagating some errors into failures And we can observe the error directly not just the results of it Invariant checkers Logging Extension of assertions to do a full-scale program state checking Inspect program state at some points during execution to ensure state correctness Of course we run invariant checkers only during testing they are too expensive to run during actual program run Very important in any serious project to trace what and how the program is doing at any given time Projects program traces into developer-friendly form Marat Akhin (SPbSPU) Software Testing /12/ / 51

47 Coverage How to measure testing? How to measure testing? OK we have ¾design for testability And we have some kind of testing in our project Question is: How do we measure software testing? We need a way to estimate testing ¾eciency yes, this is another major pain in software testing Marat Akhin (SPbSPU) Software Testing /12/ / 51

48 Coverage Four types of coverage Four types of coverage We use dierent coverage criteria to measure testing eciency There are 4 basic types of coverage: Graph based coverage Logic based coverage Input partition based coverage Syntax based coverage Paul Ammann, Je Outt, ¾Introduction to Software Testing Marat Akhin (SPbSPU) Software Testing /12/ / 51

49 Coverage Graph based coverage Graph based coverage Cover some elements of some graph corresponding to the program In other words, ¾Find a graph and cover it! Most common type of coverage due to simplicity (we're not talking about computational simplicity here) Examples include: Statement coverage Branch coverage Path coverage Data ow coverage Marat Akhin (SPbSPU) Software Testing /12/ / 51

50 Coverage Graph based coverage Statement coverage Cover every node in program's CFG (control ow graph) CFG node = basic block part of code with one entry point, one exit point and no jump instructions within it The simpliest of all statement coverage criteria In our example Single test run is enough to achieve 100% statement coverage Marat Akhin (SPbSPU) Software Testing /12/ / 51

51 Coverage Graph based coverage Branch coverage Cover every branch in program's CFG (control ow graph) Stronger than statement coverage we test all possible control ow jumps (but we do it ¾context-free ) In our example We need at least two test runs to achieve 100% branch coverage Marat Akhin (SPbSPU) Software Testing /12/ / 51

52 Coverage Graph based coverage Path coverage Cover every possible path in program's CFG (control ow graph) Much stronger than branch coverage we test all possible control ow jumps in all possible combinations (i.e., ¾context-sensitive ) Unfortunately, covering all execution paths is NP-complete task In our example We need at least four test runs to achieve 100% path coverage If we had even one back-edge in our CFG... Marat Akhin (SPbSPU) Software Testing /12/ / 51

53 Coverage Logic based coverage Logic based coverage Every conditional statement gives us two possible branches to cover But consider this one: i f ( p!= q && ( p == n u l l! p. equals ( q ))) { }... There are four logical branches to cover we cannot cover them simply by covering all branches Marat Akhin (SPbSPU) Software Testing /12/ / 51

54 Coverage Logic based coverage Logic based coverage Let's take a look at these logical branches p!= q && (p == null!p.equals(q)) Result T F T T T T - T T F F F F - - F Much stronger than path coverage Even more ¾context-sensitive than path coverage Even more ¾NP-complete than path coverage Marat Akhin (SPbSPU) Software Testing /12/ / 51

55 Coverage Input partition based coverage Input partition based coverage In a way evolution of logic based coverage criteria We have input data domain D It can be partitioned into several equivalence classes E i The equivalence classes must satisfy two main properties: E i E j =, i, j : i j (equivalence classes must be disjoint) i E i = D (equivalence classes must completely cover input data domain) Seems pretty simple, doesn't it? Marat Akhin (SPbSPU) Software Testing /12/ / 51

56 Coverage Input partition based coverage Input partition based coverage Main problem with input partitioning is ambiguity Consider the following example: Program that determines whether the le is sorted in ascending or descending order That's easy! E 1 le is sorted in ascending order E 2 le is sorted in descending order E 3 le is not sorted Marat Akhin (SPbSPU) Software Testing /12/ / 51

57 Coverage Input partition based coverage Input partition based coverage Main problem with input partitioning is ambiguity Consider the following example: Program that determines whether the le is sorted in ascending or descending order That's easy! E 1 le is sorted in ascending order E 2 le is sorted in descending order E 3 le is not sorted What about a le with only one record? That's correct partitioning fails Marat Akhin (SPbSPU) Software Testing /12/ / 51

58 Coverage Input partition based coverage Input partition based coverage Main problem with input partitioning is ambiguity Consider the following example: Program that determines whether the le is sorted in ascending or descending order That's easy! E 1 le is sorted in ascending order E 2 le is sorted in descending order E 3 le is not sorted What about a le with only one record? That's correct partitioning fails Easier to use than logic based coverage Less complex than other coverage criterias Partitioning can only be done manually in most cases outweights rst two advantages Marat Akhin (SPbSPU) Software Testing /12/ / 51

59 Coverage Syntax based coverage Syntax based coverage A ¾very other kind of coverage criteria As you all know software is usually writen in a programming language Any programming language has some syntax rules or grammar We can use that fact to test our software let's see how we can do that Marat Akhin (SPbSPU) Software Testing /12/ / 51

60 Coverage Syntax based coverage Syntax based coverage A program is a valid string in a given grammar What if we mutate this string into another one? We can get an invalid string we are not interested in them If we get another valid string it is a mutant of the original program A mutant exhibits dierent behaviour from the original program A ¾perfect test should pass only on the original program and should fail on any possible mutant (kill a mutant) Marat Akhin (SPbSPU) Software Testing /12/ / 51

61 Coverage Syntax based coverage Syntax based coverage That is our coverage criteria I.e., how many mutants a test kills This is the main idea of mutation testing A very powerful technique of estimating testing eciency Very dicult to apply by hand Very computation-intensive NP-complete at the very least Marat Akhin (SPbSPU) Software Testing /12/ / 51

62 Regression testing What kind of testing is most widely used? What kind of testing is most widely used? We have ¾design for testability We know how to estimate testing eciency And we have some kind of testing in our project What kind of testing it is? Let's answer some questions to nd the truth Marat Akhin (SPbSPU) Software Testing /12/ / 51

63 Regression testing What kind of testing is most widely used? What kind of testing is most widely used? Does a software change big and fast during its development? Usually NO We maintain, correct, adapt and modify our software most of the time Unfortunately, these small changes can break our software big time Marat Akhin (SPbSPU) Software Testing /12/ / 51

64 Regression testing What kind of testing is most widely used? What kind of testing is most widely used? Does a software change big and fast during its development? Usually NO We maintain, correct, adapt and modify our software most of the time Unfortunately, these small changes can break our software big time What do we test during development? We test modied software in order to ensure it isn't broken by our modications Wait... That is regression testing! Marat Akhin (SPbSPU) Software Testing /12/ / 51

65 Regression testing What kind of testing is most widely used? What kind of testing is most widely used? Does a software change big and fast during its development? Usually NO We maintain, correct, adapt and modify our software most of the time Unfortunately, these small changes can break our software big time What do we test during development? We test modied software in order to ensure it isn't broken by our modications Wait... That is regression testing! Regression testing is the main testing activity in any software development project or, at the very least, it should be Marat Akhin (SPbSPU) Software Testing /12/ / 51

66 Regression testing Regression testing Regression testing How does it work? 1 We modify our program P into P 2 We select tests T that need to be run on P 3 We add new tests T that test new functionality 4 We run all these tests 5 We analyze testing results Steps (2)(5) each contain some nasty pain that we have to bear Marat Akhin (SPbSPU) Software Testing /12/ / 51

67 Regression testing Regression test selection Regression test selection How can we select tests that need to be run after a small change? Conservative approach run all tests (full regression testing) Cheap approach run tests whose test requirements are related to the changes Smart approach nd parts of program that are aected by the changes and run corresponding tests Many regression test selection techniques have been developed But almost none of them found it's way to wide-spread use Marat Akhin (SPbSPU) Software Testing /12/ / 51

68 Regression testing Regression test selection Regression test selection Why is that so? We need to ensure 4 basic properties of RTS: Inclusiveness ability to include tests that are ¾modication revealing. Unsafe RTS techniques have inclusiveness less than 100% Precision ability to omit tests that are not ¾modication revealing Eciency RTS is ecient if it takes less time to determine what tests to omit than running these tests Generality ability to work in most practical situations It is very dicult to ensure all these properties at the same time Remember ¾Good. Fast. Cheap. Pick any two? Marat Akhin (SPbSPU) Software Testing /12/ / 51

69 Regression testing Maintaining regression test suite Maintaining regression test suite Regression test suite tends to grow in time and space We need a way of maintaining test suite How to decide what tests to add and what to remove? Add a new test for every part of new functionality Add a new test for every xed bug Remove tests that are duplicates of the other tests Remove tests that have never failed What do we do when a test fails? Marat Akhin (SPbSPU) Software Testing /12/ / 51

70 Regression testing Maintaining regression test suite Maintaining regression test suite There are several possible reasons of a regression test failure The software has a regression bug Must x and retest The input test values are no longer valid Must delete or update the test The expected output is no longer valid Must update the test Usually very hard to decide which situation we have We always have to consider adding additional resources vs removing excessive tests Marat Akhin (SPbSPU) Software Testing /12/ / 51

71 Regression testing Running regression tests Running regression tests That's simple take all the selected tests and run them, right? Not quite what order do we run them in? Does it matter? Yes, it does: We want to get testing results ASAP That is why we have to prioritize tests: Dierent heuristics Coverage-based prioritization History-based selection Marat Akhin (SPbSPU) Software Testing /12/ / 51

72 Resume Summary Summary Now you have a very basic glimpse of: What software testing is How it interacts with software development What are the sources of software testing pain and how to overcome them There is one piece of advice left to share with you ¾Those who cannot remember the past are condemned to repeat it George Santayana Marat Akhin (SPbSPU) Software Testing /12/ / 51

73 Resume Main rule of software testing Main rule of software testing It is very simple: Either do software testing right by the book or don't do it at all Yes, it is a ¾win or lose situation If you do software testing less than 100% you're wearing rose-coloured glasses You have a fake feeling of safety You think that you ensure good software quality You imagine you're a very good developer Sometimes it's better not to test software at all Marat Akhin (SPbSPU) Software Testing /12/ / 51

74 Resume Thank you for your time and patience Thank you for your time and patience Marat Akhin (SPbSPU) Software Testing /12/ / 51