Automated Generation of Adaptive Test Plans for Self- Adaptive Systems. Erik Fredericks and Be'y H. C. Cheng May 19 th, 2015

Transcription

1 Automated Generation of Adaptive Test Plans for Self- Adaptive Systems Erik Fredericks and Be'y H. C. Cheng May 19 th, 2015

2 Motivation Run- 9me tes9ng provides assurance for self- adap9ve systems (SAS) An SAS can experience uncertainty, possibly rendering test cases created at design 9me irrelevant Proteus ensures that test suites and test cases remain relevant throughout SAS execu9on

3 Agenda Background Proteus approach Case study Discussion Impact of run- 9me tes9ng Discussion Related work

4 Background Remote data mirroring Goal- oriented requirements engineering SoQware tes9ng

5 Remote Data Mirroring Application RDM [Veitch2003,Keeton2004] provides: Data protec9on Prevents data loss and maintains availability Stores data in physically remote loca9ons Represented as an SAS

6 Remote Data Mirroring Application

7 Network Connections

8 Dropped Message

9 Disrupted Connection

10 Reconfiguration

11 Goal- Oriented Requirements Engineering (A) Maintain [DataAvailable] (B) Maintain [Operational Costs Budget] (D) Achieve [Measure Network Properties] (E) Achieve [Minimum Num Links Active] (F) Achieve [Network Partitions == 0] (J) (K) (L) (M) (N) Achieve Achieve Achieve Achieve Achieve [Cost [Activity [LossRate [Workload [Capacity Measured] Measured] Measured] Measured] Measured] (O) Achieve [Link Deactivated] (P) Achieve [Link Activated] Link Sensor RDM Sensor Network Actuator Par0al KAOS [Dardenne1993,vanLamsweerde2009] goal model of RDM

12 Goal- Oriented Requirements Engineering Non- invariant (A) Maintain [DataAvailable] (B) Maintain [Operational Costs Budget] Invariant Requirement (D) Achieve [Measure Network Properties] (E) Achieve [Minimum Num Links Active] (F) Achieve [Network Partitions == 0] (J) (K) (L) (M) (N) Achieve Achieve Achieve Achieve Achieve [Cost [Activity [LossRate [Workload [Capacity Measured] Measured] Measured] Measured] Measured] (O) Achieve [Link Deactivated] (P) Achieve [Link Activated] Link Sensor Agent RDM Sensor Network Actuator

13 Utility Functions utility Goal_B (A) Maintain [DataAvailable] utility Goal _ B =1.0 operational_cost budget (B) Maintain [Operational Costs Budget] (D) Achieve [Measure Network Properties] (E) Achieve [Minimum Num Links Active] (F) Achieve [Network Partitions == 0] (J) (K) (L) (M) (N) Achieve Achieve Achieve Achieve Achieve [Cost [Activity [LossRate [Workload [Capacity Measured] Measured] Measured] Measured] Measured] (O) Achieve [Link Deactivated] (P) Achieve [Link Activated] Link Sensor RDM Sensor Network Actuator

14 Software Testing Requirements- based tes9ng Validate that system under test is sa9sfying requirements [Myers2011,IEEE2010] Regression tes9ng Re- validate system following major change to soqware [Myers2011,IEEE2010]

15 Software Testing Terminology Test case Single test to assess all or a por9on of a requirement Test specifica9on Set of all possible test cases derived for a soqware system Test suite Subset of test cases from the test specifica9on Typically derived to be executed under a par9cular opera9ng context Test Specifica0on TC1 TC2 TC3 TC4 TC5 TC6 TC7 TC8 TC9 TC10 Test Suite 1 TC1..TC5 TC6 TC7 TC8 Test Suite 2 TC1..TC5 TC6 TC8 TC10

16 Software Testing Terminology Test case Test Suite 1 TC1..TC5 Single test to assess all or a por9on TC6 of a requirement TC7 Test specifica9on Set TC8 of all possible test cases derived for a soqware system Test Suite 2 Test TC1..TC5 suite Subset TC6 of test cases from the test specifica9on Typically TC8 derived to be TC10 executed under a par9cular opera9ng context Test Specifica0on TC1 TC2 TC3 TC4 TC5 TC6 TC7 TC8 TC9 TC10 Invariant Non- invariant

17 Proteus Approach Requirements- driven approach for managing run- 9me tes9ng Defines an adap0ve test plan for each SAS configura9on Each configura9on corresponds to a par9cular set of environmental condi9ons, or opera'ng context Performs a tes9ng cycle during each 9mestep of SAS execu9on

18 Adaptive Test Plans ATP 1 Network loss rate Data mirror capacity Sensor fuzz ATP 2 ATP n OC 1 OC 2 OC n SAS Config. 1 SAS Config. 2 SAS Config. n

19 Adaptive Test Plans Comprises all test suites for given opera9ng context ATP 1 ATP 2 ATP n OC 1 OC 2 OC n SAS Config. 1 SAS Config. 2 SAS Config. n

20 Automa9cally derived by Defined by test engineer Proteus Adaptive Test Plans ATP 1 TS 1.0 TS1.1 TS1.3 TS1.2 TS1.4 ATP 2 ATP n OC1 OC2 OCn SAS Config. 1 SAS Config. 2 SAS Config. n

21 Test Case Activation State Test cases within test suite have an ac9va9on state: ACTIVE: Executed when current test suite is performed INACTIVE: Not executed when current test suite is performed N/A: Not executed, as it is not relevant to current opera9ng context Default test suite (TS k.0 ): Relevant to opera9ng context: ACTIVE Irrelevant test cases labeled: N/A

22 Testing Cycle Tes9ng cycle at each step of SAS execu9on 1. Execute default test suite 2. Analyze test results 3. Perform fine- grained test case parameter adapta9on 4. Perform coarse- grained test suite adapta9on 5. Determine if cycle is complete 1. If complete: halt tes9ng 2. If not complete: 1. Execute intermediate test suite

23 Test Case Fitness Test case fitness (relevance) is defined as: relevance TC =1.0 value measured value expected value expected For example: High relevance to environment Low relevance to environment Test case expected value = 0.50 Test case expected value = 0.50 Test case measured value = 0.45 Test case measured value = 0.01 Fitness = 0.90 Fitness = 0.02

24 Results Analysis Each test case is correlated to at least one goal for valida9on Test result validated against u9lity value True posi0ve Test case relevance = [Threshold, 1.0] U9lity value(s) > 0.0 True nega0ve Test case relevance = [0.0, Threshold) U9lity value(s) = 0.0 False posi0ve Test case relevance = [Threshold, 1.0] U9lity value(s) = 0.0 False nega0ve Test case relevance [0.0, Threshold) U9lity value(s) > 0.0 No ac9on necessary Error detected in SAS, perform reconfigura9on Error detected in both SAS and test case, adapt both Error detected in test case, adapt test case

25 Fine- grained Adaptation Veritas [Fredericks2014.SEAMS] Adapts non- invariant false posi9ve and false nega9ve test cases Online evolu9onary algorithm Searches for a be'er test case expected value Addresses system or environmental uncertainty for each opera9ng context Data mirror capacity TC TC1 4.65

26 Coarse- grained Adaptation Dynamically generate test suites based on test results Test Suite 1.0 TC1..TC5 TC6 TC7 TC8 TC9 Invariant True posi9ve True nega9ve False posi9ve False nega9ve Test Suite 1.1 TC1..TC5 TC7 TC8 TC9

27 End of Testing Cycle Tes9ng cycle terminates when: New SAS configura9on is invoked New tes9ng cycle ini9ated All test cases result in true posi9ves If the cycle con9nues, then the dynamically- generated test suite is executed instead of the default test suite

28 Case Study Simulated RDM network [15, 30] data mirrors [100, 200] data messages 300 9mesteps Uncertainty at each 9mestep: Unpredictable network link failures Randomly dropped or delayed messages Noise applied to data mirror sensors / network links

29 Case Study Test specifica9on: 36 test cases 7 invariant [precluded from adapta9on] 29 non- invariant [can be adapted] Compared Proteus adap9ve test plans to a manually- derived Control test plan Control test suite: All test cases from test specifica9on relevant to each opera9ng context

30 Proteus and Veritas Executed false posi9ve test cases, i.e., test case relevance = [Threshold, 1.0], u0lity value = 0.0 Executed false nega9ve test cases, i.e., test case relevance = [0.0, Threshold), u0lity value > Cumulative Number of False Positives Cumulative Number of False Negatives Control Proteus Control Proteus

31 Experimental Results Executed irrelevant test cases, i.e., test case relevance = 0.0 Total number of executed test cases Cumulative Number of Irrelevant Test Cases Cumulative Number of Executed Test Cases Control Proteus Control Proteus

32 Case Study Discussion Adap9ve tes9ng provided by Proteus framework supported by Veritas Test suites and test cases remain relevant to changing environmental condi9ons Reduces number of irrelevant test cases executed at run 9me

33 Impact of Run- Time Testing Analyzed impact of our framework: Execu9on 9me Memory overhead Requirements sa9sficement Number of reconfigura9ons

34 Impact Study Tested three states: (S1): All run- 9me tes9ng ac9vi9es enabled Proteus+Veritas enabled (S2): All run- 9me tes9ng ac9vi9es disabled Proteus+Veritas disabled (S3): Run- 9me tes9ng framework removed Proteus+Veritas code / data structures removed from implementa9on

35 Impact Execu9on 9me Measured total execu9on 9me of RDM simula9on (S1) (S2) (S3) Average execu9on 9me (seconds)

36 Impact Execu9on 9me Measured total execu9on 9me of RDM simula9on (S1) (S2) (S3) Average execu9on 9me (seconds) Significant (p < 0.05)

37 Impact Execu9on 9me Measured total execu9on 9me of RDM simula9on (S1) (S2) (S3) Average execu9on 9me (seconds) Memory footprint Measured maximum memory usage of RDM simula9on (S1) (S2) (S3) Average memory usage (megabytes)

38 Impact Execu9on 9me Measured total execu9on 9me of RDM simula9on (S1) (S2) (S3) Average execu9on 9me (seconds) Memory footprint Not significant (p > 0.05) Measured maximum memory usage of RDM simula9on (S1) (S2) (S3) Average memory usage (megabytes)

39 Impact Requirements sa9sficement Calculated average u9lity value over simula9on (S1) (S2) (S3) Average u9lity value

40 Impact Requirements sa9sficement Calculated average u9lity value over simula9on (S1) (S2) (S3) Average u9lity value Not significant (p > 0.05)

41 Impact Requirements sa9sficement Calculated average u9lity value over simula9on (S1) (S2) (S3) Average u9lity value Number of reconfigura9ons Averaged number of triggered RDM reconfigura9ons (S1) (S2) (S3) Average number of reconfigura9ons

42 Impact Requirements sa9sficement Calculated average u9lity value over simula9on (S1) (S2) (S3) Average u9lity value Number of reconfigura9ons Not significant (p > 0.05) Averaged number of triggered RDM reconfigura9ons (S1) (S2) (S3) Average number of reconfigura9ons

43 Discussion of Testing Impact Run- 9me, adap9ve tes9ng only significantly impacts RDM in terms of execu9on 9me Exploring paralleliza9on strategies to reduce 9me impact While not significant, a clear difference in mean u9lity values exist in requirements sa9sficement Timing of measurement causes sampling 9mes to be slightly different

44 Related Work Search- based so^ware tes0ng Techniques such as evolu9onary computa9on, hill climbing, simulated annealing used for different tes9ng approaches in model tes9ng [Harman2009], regression tes9ng [Harman2012], and structural tes9ng [McMinn2011] EvoSuite [Fraser2011] and Nighthawk [Andrews2011] are evolu9onary frameworks for genera9ng test suites and instan9a9ng unit tests Veritas uses a run- 9me evolu9onary algorithm, whereas the other techniques focus on design 9me search Run- 0me tes0ng Implemented using reinforcement learning [Veanes2006], recording & replaying [Tsai1990], and Markov modeling [Filieri2011] approaches Veritas combines evolu9onary search for test parameters with u9lity- based valida9on Proteus maintains relevance of test suites as condi9ons change

45 Related Work Test suite genera0on Requirements specifica9on used to generate formal grammars [Bauer1979] Proteus generates new test suites based upon a pre- defined default test suite and executes based on monitored condi9ons Ar9ficial intelligence used to automa9cally generate test plans for graphical user interfaces [Memon2001] Proteus analyzes monitoring informa9on to select appropriate test suite Test case selec0on and priori0za0on Select a representa9ve set of test cases and priori9ze their execu9on [Harman2009] Proteus selects and executes tests at run 9me Tropos [Nguyen2008] uses agent- based randomized tes9ng to validate mul9- agent systems Proteus generates test suites targeted towards specific DAS opera9ng contexts

46 Acknowledgements NSF BEACON Center ( center.org) NSF grants CCF , DBI , CNS , CNS Ford Motor Company General Motors

47 Q&A

48 References [Veitch2003] M. Ji, A. Veitch, and J. Wilkes, Seneca: Remote mirroring done write, in USENIX 2003 Annual Technical Conference. Berkeley, CA, USA: USENIX Associa9on, June 2003, pp [Keeton2004] K. Keeton, C. Santos, D. Beyer, J. Chase, and J. Wilkes, Designing for disasters, in Proc. of the 3rd USENIX Conference on File and Storage Technologies. Berkeley, CA, USA: USENIX Associa9on, 2004, pp [Dardenne1993] A. Dardenne, A. Van Lamsweerde, and S. Fickas, Goal- directed re- quirements acquisi9on, Science of computer programming, vol. 20, no. 1, pp. 3 50, [vanlamsweerde2009] A. van Lamsweerde, Requirements Engineering: From System Goals to UML Models to SoLware Specifica'ons. Wiley, [degrandis2009] P. degrandis and G. Vale'o, Elicita9on and u9liza9on of applica9on- level u9lity func9ons, in Proc. of the 6th Interna'onal Conference on Autonomic Compu'ng, ser. ICAC 09. ACM, 2009, pp [Ramirez2011] A. J. Ramirez and B. H. C. Cheng, Automa9cally deriving u9lity func9ons for monitoring soqware requirements, in Proc. of the 2011 Interna'onal Conference on Model Driven Engineering Languages and Systems Conference, Wellington, New Zealand, 2011, pp [Walsh2004] W. E. Walsh, G. Tesauro, J. O. Kephart, and R. Das, U9lity func9ons in autonomic systems, in Proc. of the First IEEE Interna'onal Conference on Autonomic Compu'ng. IEEE Computer Society, 2004, pp [Myers2011] G. J. Myers, C. Sandler, and T. Badge', The art of solware tes'ng. John Wiley & Sons, [IEEE2010] IEEE, Systems and soqware engineering vocabulary, ISO/IEC/IEEE 24765:2010(E), pp , Dec 2010.

49 References [Fredericks2014.SEAMS] E. M. Fredericks, B. DeVries, and B. H. C. Cheng, Towards run- 9me adapta9on of test cases for self- adap9ve systems in the face of uncertainty, in Proceedings of the 9th Interna'onal Symposium on SoLware Engineering for Adap've and Self- Managing Systems, ser. SEAMS 14, [Harman2009] M. Harman, S. A. Mansouri, and Y. Zhang, Search based soqware engineering: A comprehensive analysis and review of trends techniques and applica9ons, Department of Computer Science, King s College London, Tech. Rep. TR , [Fraser2011] Gordon Fraser and Andrea Arcuri. Evosuite: automa9c test suite genera9on for object- oriented soqware. In Proc. of the 19th ACM SIGSOFT symposium and the 13th European conference on Founda9ons of soqware engineering, ES- EC/FSE 11, pages , Szeged, Hungary, ACM. [Andrews2011] James H. Andrews, Tim Menzies, and Felix C.H. Li. Gene9c algorithms for randomized unit tes9ng. IEEE Trans. on SoQware Engineering, 37(1):80 94, January [Veanes2006] Margus Veanes, Pritam Roy, and Colin Campbell. Online tes9ng with rein- forcement learning. In Formal Approaches to SoQware Tes9ng and Run9me Verifica9on, pages Springer, [Tsai1990] J.J.- P. Tsai, K.- Y. Fang, Horng- Yuan Chen, and Yao- Dong Bi. A noninter- ference monitoring and replay mechanism for real- 9me soqware tes9ng and debugging. SoQware Engineering, IEEE Transac9ons on, 16(8): , [Filieri2011] Antonio Filieri, Carlo Ghezzi, and Giordano Tamburrelli. Run- 9me efficient probabilis9c model checking. In Proc. of the 33rd Interna9onal Conference on SoQware Engineering, pages , Waikiki, Honolulu, Hawaii, USA, ACM.