Software Engineering. Zheng Li( 李征 ) Jing Wan( 万静 )

Transcription

1 Software Engineering Zheng Li( 李征 ) Jing Wan( 万静 )

2 作业 Automatically test generation 1. 编写一个三角形程序, 任意输入三个整数, 判断三个整形边长能否构成三角形, 如果是三角形, 则判断它是一般三角形 等腰三角形或等边三角形, 并输出三角形的类型 2. 画出程序的 CFG, 计算圈复杂度 3. 设计一组测试用例满足测试准则

3 Problem: example User inputs three sides of a triangle (a, b, c). Which type is the triangle? Requirements: IF a<=0 b<=0 c<=0 -> input incorrect IF (a+b<c a+c<b b+c<a) -> sides not forming a triangle IF a==b a==c b ==c -> isoceles Kui a==b & b==c -> eqilateral Other -> scalene What strategy? -> what data?

4 Input validation automation The Concept side of a triangle equivalence partitions and boundary values Normal: ]0; [ Erroneous: ]- ; 0[, missing values Border values: {0} For testing the Input validation functionality, pick a random value from each equivalence partition for each side: P(-1, 2, 3), P(1, -2, 3), P(1, 2, -3) Same with boundary values P(0, 1, 2), P(1, 0, 2), P(1, 2, 0) Input validation with normal values P(1, 2, 3)

5 What about other requirements? If input values are dependent and that affects output, random values cannot be used! we may not be able to find needed values with random generation if (a+b<c a+c<b b+c<a) -> sides don t form a triangle We must use specification-based (Black-box) or program-based (White-box) test data generation

6 研讨班课程 Automatically test generation 1. 采用随机法生成测试用例, 要求实现语句覆盖, 统计随机生成测试用例的个数

7 AUTOMATICALLY TEST GENERATION

8 Automatically test generation Exhaustive Testing Random Testing Intelligent Approach (SBST)

9 What is random testing The principle of random testing is simple and can be described as follows: 1. For each input parameter, generate a random but legal value. 2. Apply the full set of inputs to the SUT 3. Register the result and go back to step 1.

10 Generating Test Cases Randomly bool isequal(int x, int y) { if (x = y) z := false; else z := false; return z; } If we pick test cases randomly it is unlikely that we will pick a case where x and y have the same value If x and y can take 2 32 different values, there are 2 64 possible test cases. In 2 32 of them x and y are equal probability of picking a case where x is equal to y is 2-32

11 Further thinking on random How many times you run the random program? True random generators VS pseudo-random generators Proportional sampling Uniform distribution

12 Block failure pattern

13 Strip failure pattern

14 Point failure pattern

15 Chen s observation Inputs that are close to each others in the input domain tends to go through the same path. Thus, in order to find most of the errors, we should spread the test cases as much as possible. This approach is called Adaptive Random Testing.

16 Adaptive Random Testing (ART) Choose test cases at random One at the time Add to current set G (initially empty) But when adding new one: 1. Sample Z test cases at random 2. Choose from Z the one that is most diverse (z) compared to current set G. 3. Add z to G

17 ART algorithms 1. Selection of the best candidate as the next test case from a set of candidates. This approach first generates a set of random inputs as candidates from which the best candidate, as defined against set criteria, is selected as the next test case. 17

18 ART algorithms 2) Exclusion In each round of test case generation, this approach first defines an exclusion region around each already executed test case. Random inputs are generated one by one until one input is outside all exclusion regions of the already executed test cases, and then this input is selected as the next test case. 18

19 ART algorithms 3) Partitioning This approach uses the information about the location of already executed test cases to divide the input domain into partitions, and then to identify a partition as a designated region from which the next test case will be generated. 19

20 ART algorithms 4) Test profiles This approach uses a specially designed test profile which is able to achieve an even spreading of test cases over the input domain. Dynamic adjustment of the test profile during testing is required in this approach. 20

21 ART algorithms 5) Metric-driven Distribution metrics, such as discrepancy and dispersion, are normally used to measure the degree of even distribution for a set of points. This approach uses the distribution metrics as selection criteria to select new test cases such that a more even distribution of the resultant test cases could be obtained. 21

22 Terminologies Failure rate: failure causing inputs all possible inputs Failure pattern: distributions and geometry of the failure-causing inputs Efficiency: the computation time required Effectiveness: the fault detection capability

23 Terminologies(Contd ) F-measure: expected number of test cases required to detect the first failure P-measure: probability of detecting at least one failure; E-measure: expected number of failures detected. 23

24 Effectiveness in terms of F-measure ART is better than RT with respect to the F-measure. Because, even spreading of test cases originates from the objective of hitting the contiguous failure regions using fewer test cases. 24

25 Effectiveness in terms of P-measure By definition, the P-measure for RT or ART is a function of the size of the test set Size of the test set has significant impact on the returned values of the P-measure, hence F-measure is more appropriate. All studies have consistently shown that ART outperforms RT with respect to the P-measure. 25

26 Effectiveness in terms of E-measure With regard to the E-measure, PSS, Proportional Sampling Strategy (ART algorithm) and RT have been theoretically proved 26

27 Effectiveness in terms of time ART required an average of 1.6 times the amount of time required by RT to detect the first failure. 27

28 Evolutionary Testing

29 Evolutionary Testing Insertion Mutation Fitness evaluation Recombination End? Selection

30 Search Algorithm - GA Genetic Algorithm Directed search algorithms based on the mechanics of biological evolution Developed by John Holland, University of Michigan (1970 s) To understand the adaptive processes of natural systems To design artificial systems software that retains the robustness of natural systems

31 Components of a GA A problem to solve, and... Encoding technique (gene, chromosome) Initialization procedure (creation) Evaluation function (environment) Selection of parents (reproduction) Genetic operators (mutation, recombination) Parameter settings (practice and art)

32 Simple Genetic Algorithm { } initialize population; evaluate population; while TerminationCriteriaNotSatisfied { select parents for reproduction; perform recombination and mutation; evaluate population; }

33 GA Cycle of Reproduction Initialize population Compute fitness Judge whether meet the rules or not? Best individuals Selection Begin End Crossover Mutation

34 A Simple Example The Traveling Salesman Problem: Find a tour of a given set of cities so that each city is visited only once the total distance traveled is minimized

35 Representation Representation is an ordered list of city numbers known as an order-based GA. 1) London 3) Dunedin 5) Beijing 7) Tokyo 2) Venice 4) Singapore 6) Phoenix 8) Victoria CityList1 ( ) CityList2 ( )

36 Crossover Crossover combines inversion and recombination: * * Parent1 ( ) Parent2 ( ) Child ( ) This operator is called the Order1 crossover.

37 Mutation Mutation involves reordering of the list: * * Before: ( ) After: ( )

38 TSP Example: 30 Cities

39 y Solution i (Distance = 941) TSP30 (Performance = 941) x

40 y Solution j (Distance = 800) TSP30 (Performance = 800) x

41 y Solution k (Distance = 652) TSP30 (Performance = 652) x

42 y Best Solution (Distance = 420) TSP30 Solution (Performance = 420) x

43 Evolutionary Testing Insertion Mutation Test cases Execution Fitness evaluation Recombination Monitoring Selection End?

44 CI Computational Intelligence 计算智能 : 以生物进化的观点认识和模拟智能 智能的产生 : 是在生物的遗传 变异 生长以及外部环境的自然选择中产生的 在优胜劣汰的过程中, 适应度高的结构被保存下来, 智能水平也随之提高 计算智能就是基于结构演化的智能

45 CI Computational Intelligence 计算智能算法 神经计算 : 人工神经网络算法 模糊计算 : 模糊逻辑 进化计算 : 遗传算法 蚁群优化算法 粒子群优化算法 免疫算法 分布估计算法 Memetic 算法, 模拟退火算法 禁忌搜索算法等

46 计算智能与生物智能 痛觉 生物 : 一种碳基生命, 痛觉是为了避开伤害 这些伤害可能会对生物的生存产生重大的影响, 伤害无法修复 计算智能 : 价值损失 + 时间损失

47 CI Computational Intelligence 计算智能主要解决的问题 : 可计算性理论 (Computability Theory) P 类 (Polynomial) 问题 如果一个问题可以找到一个能在多项式的时间里解决它的算法, 那么这个问题就属于 P 问题 NP 类 (None-deterministic Polynomial) 问题 NP 类问题并不是 P 类问题的对立面 ( 或非 P 类问题 ), 而是指可以在多项式的时间里验证 ( 而非获得 ) 一个解的问题

48 Optimising Structural Testing

49 Program Coverage TARGET control flow graph of a program containing a structure we want to cover

50 Fitness evaluation The test data executes the wrong path TARGET

51 Analysing control flow The outcomes at key decision statements matter. TARGET These are the decisions on which the target is control dependent

52 Approach Level = 2 = 1 minimisation TARGET = 0

53 Analysing predicates Approach level alone gives us coarse values a = 50, b = 0 a = 45, b = 5 a = 40, b = 10 a = 35, b = 15 a = 30, b = 20 a = 25, b = 25 getting closer to being true

54 Branch distance Associate a distance formula with different relational predicates a = 50, b = 0 branch distance = 50 a = 45, b = 5 branch distance = 40 a = 40, b = 10 branch distance = 30 a = 35, b = 15 branch distance = 20 a = 30, b = 20 branch distance = 10 a = 25, b = 25 branch distance = 0 getting closer to being true

55 Putting it all together Fitness = approach Level + normalised branch distance true if a >= b false true if b >= c true if c >= d false TARGET false TARGET MISSED Approach Level = 2 Branch Distance = b - a TARGET MISSED Approach Level = 1 Branch Distance = c - b TARGET MISSED Approach Level = 0 Branch Distance = d - c normalised branch distance between 0 and 1 indicates how close approach level is to being penetrated

56