LCCI (Large-scale Complex Critical Infrastructures)

Transcription

1

2 LCCI (Large-scale Complex Critical Infrastructures) 1 LCCIs are Internet-scale constellations of heterogeneous systems glued together into a federated and open system by a data distribution middleware. The shift towards Internet is considered a necessary step to overcome the limitations of the monolithic and closed architectures used traditionally to build critical systems (e.g., SCADA architectures). Real world example is the novel framework for Air Traffic Management (ATM) that EuroCONTROL is developing within the SESAR EU Joint Undertaking.

3 LCCI (Large-scale Complex Critical Infrastructures) 2 New challenges rise from LCCIs that push the frontiers of current technologies. Data distribution task becomes crucial and has to be: Reliability: deliveries have to be guaranteed despite failures may happen; Timeliness: messages must reach their destinations at the right time, without breaking temporal constraints; Scalability: performance is affected neither by the time nor by the LCCI size. The challenge is to find the best data distribution paradigm able to meet the aforementioned requirements.

4 Outline of SWIM concept 3 SWIM (System Wide Information Management) aims to establish a seamless interoperability among heterogeneous ATM stakeholders: common data representation; coherent view on current ATM information (e.g. Flight Data, Aeronautical Data, Weather). It may be seen as a common data/service bus on which systems having to interoperate are connected. Close in spirit to a middleware solution for LCCI.

5 SWIM prototype 4 The prototype (named SWIM-BOX ) has been conceived as a sort of Gateway/Mediator across legacy applications: Completely distributed architecture; Designed using a domain based approach (Flight, Surveillance, etc); Implemented using a standard based approach; Well known data and information models (e.g. ICOG2); Standard technologies (Web Services, EJB, DDS); DDS-compliant middleware for sharing data. Legacy A SWIM- BOX SWIM Network SWIM- BOX Legacy B Adapte r A Adapte r B Legacy site Common Infrastructure Legacy site

6 Some challenges 5 How subsystems (as COTS) involved into LCCI impacts on its dependability? What are the effects on LCCI if DDS-compliant middleware is invoked with erroneous inputs? Robustness testing provides answers to these questions: Help vendors evaluating their implementations; Help clients selecting several solutions. Tests cost reduction automating tests procedure. Automating tests results classification.

7 Our goal Assessing the robustness middleware What does robustness mean? 6 of DDS-compliant The degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions [IEEE Std ]. Dependability with respect to external faults, which characterizes a system reaction to a specific class of faults [Avizienis 04]. Robustness testing features: Only the system interface has to be known; Source code is not needed (black-box approach); Injecting exceptional input through API; Do not alter data and structure" internally; Select carefully inputs and stressful conditions that cause the activation of faults representative of actual situations.

8 Robustness Testing Approaches 7 Robustness testing: stressing the public interface of the application/system/api with invalid and exceptional values: From Application To System Under Test (Top-Down); From OS to System Under Test (Bottom-UP). API called with exceptional values Application DDS Middleware OS syscall Operating System OS return with exceptional values

9 Robustness Testing Approaches 8 Robustness testing: stressing the public interface of the application/system/api with invalid and exceptional values: From Application To System Under Test (Top-Down); From OS to System Under Test (Bottom-UP). API called with exceptional values Application DDS Middleware OS syscall Operating System OS return with exceptional values

10 Robustness Testing Approaches 9 Robustness testing: stressing the public interface of the application/system/api with invalid and exceptional values: From Application To System Under Test (Top-Down); From OS to System Under Test (Bottom-UP). Workload stands for a set of valid calls. It s needed to stress each operation of the device under test. Fault model is a set of rules applied at API to expose robustness problems. Failure mode classification characterizes the behavior of the system under test while executing the workload in the presence of fault model.

11 Fault Injection: WWW dilemma 10 What to inject? Fault model -> Fault List Where to inject? At API interface level Method with higher occurrences (Method list) When to inject? At only one invocation of methods (Trigger list) Injection library Fault list Method list Trigger List Fault, Model and Trigger lists define our Injection library

12 Faults list 11 The rules list applied during the API invocation: Each method input is tested with all robustness values one for time. E.g., void replace(int a, String b).

13 Method list 12 Profiling different applications using DDScompliant middleware product: Ping-pong application; Touchstone: benchmarking framework for evaluating the performance of OMG DDS compliant implementations; SWIM-BOX. The methods occurrences have been measured for each applications: Only a limited set core of all available methods are invoked; The same occurrences distribution is noted for all applications Method list involved the methods with higher occurrences.

14 Failure mode classification 13 CRASH scale has been utilized to classify the robustness problems Catastrophic: node crashes and OS hangs, DDS provider do not deliver messages correctly. Restart: DDS provider becomes unresponsive and must be terminated by force. Abort: Abnormal termination when invoking API. Silent: Faulty submitted value doesn t rise exceptions, despite this message are or aren t transmitted. Hindering: returned error code is incorrect. Further and suitable levels have been added: non conformity: fault is not indicated as should be. DDS API analysis has been performed for results classification. Golden run has been run for each injecting value to understand the system behavior.

15 Test automation: JFault Injection Tool (JFIT) 14 Pros: Cons: Java-based implementation; No knowledge about the SUT; Run-time methods interception and values mutation: Exploiting java reflection; Monitoring status and output of the SUT. Only methods with primitive types (i.e. String, int, ) are taken into account; Off line and by hand results classification.

16 High level architecture of JFIT 15 All robustness test are carried out according with the Injection library; Controller is in charge for tests management and runs them through the Activator; Interceptor catches the methods invocation to SUT and injects, by Injector, the faults one for time Monitor records the output at Pub and Sub side. MONITOR CONTROLLER ACTIVATOR System Under Test INTERCEPTO R INJECTOR

17 Test execution stages 16 Preliminary execution of the workload without faults To understand the normal behavior Starting robustness testing DDS initialitation Workload execution Monitoring & Logging Injection phase Golden run One fault for time No faults are injected

18 Tests Results DDS middleware: OpenSplice implementation; No QoS features have been defined (Best Effort); 17 According with the failure mode classification the achieved results are as follows: no Catastrophic, Abort and Hindering problems have been evidenced: Neither node crashes and nor OS hangs; No abnormal termination when invoking API; No erroneous returned error code. 13% of robustness tests have shown Restart problems: Experiment doesn t response and must be terminated by force. 45% of robustness tests have risen Silent problems: No exception has been thrown by DDS;

19 Tests Results Faults distribution between Silent and Restart. Int faults types String faults types 18 Faults types

20 Conclusions 19 Our approach can automatically test the core set of DDS methods; A significant fraction of tests shows some robustness issues raised when exceptional values are submitted to OpenSplice APIs (e.g., large strings, or big integers); The ability to reach a consistent system state before performing fault injection makes us confident of the results.

21 Conclusions 20 Our approach can automatically test the core set of DDS methods; A significant fraction of tests shows some robustness issues raised when exceptional values are submitted to OpenSplice APIs (e.g., large strings, or big integers); The ability to reach a consistent system state before performing fault injection makes us confident of the results. Ongoing activities Testing all parameters types and not only primitive types; Automating results classification; Running tests in presence of quality of service mechanisms; Carrying out the same tests with other DDS-compliant middleware.

22 References 21 [Avizienis 04] A. Avizienis, J.C. Laprie, B. Randell, C. Landwehr. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable Secure Computing, [Koopman 02] P. Koopman. What's Wrong With Fault Injection As A Benchmarking Tool?. in Proc. DSN 2002 Workshop on Dependability Benchmarking, pp. F , Washington, D.C.,USA, [Koopman 99] Koopman P., DeVale J., Comparing the robustness of POSIX operating, Proceedings of Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing, [Johansson 07] Johansson A., Suri N., Murphy B. On the selection of Error models for OS Robustness Evaluation Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, [Miller 95] B.P. Miller et al, Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities., Technical report, 1995.

23 22

24 23 Test Scenario A receiver is waiting for messages The Transmitter sends burst of messages for a while then terminates Further details JFIT DDS middleware: OpenSplice implementation No QoS features have been defined (Best Effort) A P I in te rc e p to r A P I in je c to r Mo nito rin g Mo n ito rin g JFIT

25 Pub/Sub paradigm 24 Pub/Sub reveals effective to federate heterogeneous systems Space, time and synchronization decoupling enforce scalability Asynchronous multi-point communication good to devise cooperating systems HERALD CORBA NS GREEN JEDI HERMES SIENA DREAM JMS Among the plethora of Pub/Sub alternatives DDS exhibits better performances, higher scalability and larger set of offered QoS Widely used in large scope initiatives addressing wide area scenarios E.g., it has been investigating as the data distribution system into SESAR project through SWIM middleware infrastructure