Adaptive Fault Tolerant Systems: Reflective Design and Validation

Transcription

1 1 Adaptive Fault Tolerant Systems: Reflective Design and Validation Marc-Olivier Killijian Dependable Computing and Fault Tolerance Research Group Toulouse - France

2 2 Motivations Provide a framework for FT developers Open Flexible Dependability of both embedded and large scale distributed systems Adaptation of fault tolerance strategies to environmental conditions and evolutions Validate this framework Test Fault-injection

3 3 History Reflection for Dependability 1. Friends v1 - off-the-shelf MOP Limits: static MO, inheritance, etc. 2. Friends v2 - ad-hoc MOP / CT reflection 3. Multi-Level Reflection Validation of the platform Test of MOP based architectures Fault-injection and failure modes analysis

4 4 Outline Reflection for Dependability 1. Friends v1 - off-the-shelf MOP Limits: static MO, inheritance, etc. 2. Friends v2 - ad-hoc MOP / CT reflection 3. Multi-Level Reflection Validation of the platform Test of MOP based architectures Fault-injection and failure modes analysis

5 5 Why Reflection? Separation of concerns Non functional requirements Applications Adaptation Selection of mechanisms w.r.t. needs Changing strategies dynamically Portability/Reuse Reflective platform (relates to adaptation) Meta-level software (mechanisms)

6 6 Overall Philosophy Metalevel (metaobjects and policies) One-to-one One-to-many Dynamic association Baselevel (application objects) Wrappers API Middleware OS / kernel Comm. protocols Hardware platform Testing strategies Mechanisms Development Failure Modes analysis Assumptions Faults SWIFI Fault model

7 7 Friends v2 : A MOP on Corba MOP design Identify information to be reified and controlled MOP implementation Compile-time reflection Using CORBA facilities Prototype for illustration Architecture and basic services Fault tolerance mechanisms Preliminary performance analysis

8 8 Necessary information : integrated mechanism example Client 1-invocation 7-reply Server 4-send checkpoint 6-return Replica 2-process invocation 3-obtain checkpoint 5-apply checkpoint

9 9 Necessary information : metaobjects example Meta Stub MO Server 4-send checkpoint MO Replica 7-reply 6-return Client Stub Server Replica 1-invocation 2-process invocation 3-obtain checkpoint 5-apply checkpoint Observation invocations state capture Control method calls state restoration

10 10 Which protocol? Observation Interception Creation Destruction Invocations (In and out) State capture Links control Object/metaobject Clients/servers Reification Meta Object Object Activation Creation Destruction Invocations Control State restoration Links control Object/metaobject Clients/servers Intercession

11 11 Protocol definition Meta stub Metastub 1 Meta 2 object Metaobject Client Stub Stub Object Server 1 2 Protocol and interfaces specific to a mechanism

12 12 Using Open Compiler Meta Class Compile-time MOP Class Open Compiler Class + MOP o.foo(); interinfo TranslateMethodCall ( ReifiedInfo) {.. return NewCode; } NewCode

13 13 Architecture Services OF OF MOF MOF GS GS OF OF MOF MOF GS GS OF OF MOF MOF GS GS MS1 MS1 MO1 MO1 MO2 MO2 MOP MOP MOP C1 C1 S1 S1 S2 S2 Node 1 Node 1 ORB ORB Node 2 Node 2 Node 3 Node 3

14 14 Results A method for designing a MOP Analysis of mechanisms needs! MOP features Metaobject protocol for fault tolerance Transparent and dynamic association Automatic handling of internal state (full/partial) Portable serialization [OOPSLA 02] Smart stubs delegate adaptation to meta-stubs CORBA compliant (black-box) Some programming conventions

15 15 Lessons Learnt Generic MOP No assumption on low layers Based on CORBA features! With a platform «black-box» Language dependent Limitations external state determinism Open platform (ORB, OS and language)! Additions of new features to the MOP! Optimization of reflective mechanisms! Language level reflection still necessary FT Language ORB Runtime OS

16 16 Limits to be addressed Behavioral issues Concurrency models: Middleware level Threading and synchronization: Middleware/OS level Communication in progress: Middleware/OS level Structural/State issue Site-independent internal state : Open Languages Site-dependent internal state: Problems: Identification, handling Available means: Syscall interception, Journals and replay monitors External state Middleware level OS level " Concept of multilevel reflection

17 17 Which protocol? Observation Meta Object Interception Creation Destruction Invocations (In and out) Threading Synchronization Communication State capture Internal objects Site-dependent objects External objects (MW+OS) Links control Object/metaobject Clients/servers Control Activation/control Creation Destruction Invocations Threading Synchronization Communication State restoration Internal objects Site-dependent objects External objects (MW+OS) Links control Object/metaobject Clients/servers Reification Object Intercession

18 18 Which Platform? C S Universal VM for Distributed Objects Middleware Middleware Middleware Language Support Language Support OS OS OS Hardware Hardware Hardware Fault-Tolerance

19 19 Which Platform? This one? But difference between OS/MW LS/MW? C S Universal VM for Distributed Objects Middleware Middleware Middleware Language Support Language Support OS OS OS Hardware Hardware Hardware Fault-Tolerance

20 20 Which Platform? Or this one? C S Universal VM for Distributed Objects Middleware Middleware Middleware Language Support Language Support OS OS OS Hardware Hardware Hardware Fault-Tolerance

21 21 Which Middleware? FT needs to be aware of everything (potentially) C S Universal VM for Distributed Objects Under-ware Under-ware Fault-Tolerance Hardware Hardware Hardware

22 22 Which Middleware? FT needs to be aware of everything (potentially) but how? Reflective languages Reflective middleware Reflective OS Fault-Tolerance A lot of different concepts to manipulate

23 23 Multi-level level Reflection mono-level meta-models aggregation multi-level meta-model Fault-Tolerance Self-contained, integrated, meta-interface

24 24 Multilevel Reflection Apply reflection to a complete platform Application, Middleware, Operating System Consistent view of the internal entities/concepts Transactions, stable storage, assumptions Memory, data, code Objects, methods, invocations, servers, proxies Threads, pipes, files Context switches, interrupts Define metainterfaces and navigation tools Which metainterface (one per level? Generic?) Consistency # metamodel

25 25 Different Aspects Intra-level information Necessary for FT Efficiency (lowest possible? Same concepts at levels?) Inter-level information ML management (inter-level coupling) Adaptation Concepts/levels navigation mono-level meta-models aggregation multi-level meta-model Fault-Tolerance Self-contained, integrated, meta-interface

26 26 Requirements of FT- Mechanisms? Non determinism of scheduling/execution time!interlevel interactions mostly asynchronous Trend: Leverage know-how on FT asynch. distributed sys.! Causality tracking/ monitoring of non-determinism is needed.! State capture/ recovery at appropriate granularity is needed.! (?) q p r p q r [Kasbekar99, Basile02] P t 1 o 2 o t 2 1 P o 1 t 2o2 t 1

27 27 Inter-Level Coupling (I) A Level = 1..n COTS = A set of interfaces = Concepts Primitives / base entities (keywords, syscalls, data types, ) Rules on how to use them (concepts, base entities, rules) = programming model Very broad notion (includes programming languages) Self contained Base entities a-tomic within that programming model Can t be split in smaller entities within the programming model. Implemented by more elementary entities within the component. Implementation is internal hidden to component user.

28 28 Inter-Level Coupling (II) CORBA : Location transparent object method invocation A CORBA request = aggregation Communication medium ( pipes, sockets, ) Local control flow ( POSIX threads, Java threads, LWP, ) Global control flow abstraction transparent interaction Appli. Mw. observation level client client CORBA interaction thread signal socket mutex server server composite interaction chain

29 29 Inter-Level Coupling (III) Within a COTS : Coupling between emerging entities of next upper level and implementation entities of lower levels Structural coupling relationships ( abstraction mappings ) translation / aggregation / multiplexing / hiding Dynamic coupling relationships ( interactions ) creation / binding / destruction / observation / modification Appli. Mw. Level N+1 Level N

30 Extracting Coupling in CORBA (I) 30 Appli. Mw. observation level client client CORBA interaction thread signal socket mutex server server

31 (II) Extracting Coupling in CORBA 31 Behavioral model of connection oriented Berkeley sockets as seen by the middleware programmer * accept call socket close shutdown idle accepting connections bound unbound accept return bind ; listen * new socket call to recv idle send return from recv waiting for reception

32 Extracting Coupling in CORBA Object Creation Thread Creation (III) 32 Method Invocation Socket API bind listen accept recv send shutdown ** Dynamic coupling between CORBA invocations and the socket API

33 33 FT + Inter-Level Coupling Top-down observation & control State capture Monitoring of non-determinism (I) System's Functional Interface Application Layer L A Abstraction Level Lev n+1 Executive Layer L n+1 Abstraction Level Lev n Executive Layer L n Abstraction Level Lev n-1

34 34 FT + Inter-Level Coupling Bottom-up observation & control Fault propagation analysis / confinement (II) Rollback propagation / state recovery System's Functional Interface Application Layer L A Abstraction Level Lev n+1 Executive Layer L n+1 Abstraction Level Lev n Executive Layer L n Abstraction Level Lev n-1

35 35 Meta-filters All the information is not always necessary Specific mechanisms need specific info Mechanisms can change over time Need a way to dynamically filter Efficiency Don t reify unnecessary things Have hooks ready but passified + subscriptions Meta-filters implementation Simple boolean matrices Code-injection techniques

36 36 Current & Future Work on MLR Still some work on ORB/OS analysis Implementation a la carte : several «flavours» Radical style # full metamodel from scratch or based on modified open-source components Middle-Way based on available reflective components + wrappers EZ way wrapped COTS # limited metamodel Evaluate the benefits on mechanisms Efficiency /ad-hoc /language level reflection Evolution of non-funtionnal requirements/asumptions Environmental evolution Validation Rigourous testing stategies for reflective/adaptive systems Characterization by various ad-hoc fault injection techniques

37 37 Adaptive Fault Tolerant Systems Part II- Testing Reflective Systems Reflection 00 - DSN 01- IEEE ToC 03 Ruiz, Fabre, Thevenod, Killijian Dependable Computing and Fault Tolerance Research Group Toulouse - France

38 38 Motivations for testing MOPs In reflective architectures the MOP is the corner stone FT completely rely on the reflective mechanisms Very little work has been done Few on formal verification None on testing Validation of the FT architectures Test of the underlying platform Fault-injection

39 39 Testing Reflective Systems 1. Test order definition (reification reification,, intercession, introspection) 2. Test objectives for each testing level 3. Conformance checks for each testing level 4. Test environments

40 40 Testing MOPs TL0.. Testing preceding the MOP activation TL1.. Reification mechanisms TL2.. Behavioral intercession mechanisms TL3.. Introspection mechanisms TL4. Structural intercession mechanisms

41 41 Incremental Test Order TL0. implementation dependent TL1.. Reification mechanisms TL2.. Behavioral intercession mechanisms TL3.. Introspection mechanisms TL4. Structural intercession mechanisms

42 42 TL1: Reification (behavioral observation) 3 Oracle Comparison: - Invoked method - Parameter values - Output values 6 metaobject metaobject - Observation - Method execution is simulated by generating a random value for each output parameter object test driver service server

43 43 Oracle TL2: Behavioral intercession Server traces are checked according to the data supplied by the test driver 8 (behavioral control) metaobject metaobject - Reified information is systematically delivered to the server object - Output values are returned to the test driver test driver 4 service Behavioral intercession 3

44 44 TL3: Introspection (structural observation) 1 metaobject metaobject metacontrol 3 test driver service 2 Behavioral intercession 4 Introspection State 5 6 Is the introspected state consistent with the initialization state? Oracle 7

45 45 TL4: Structural intercession (structural control) metaobject metaobject metacontrol test driver 1-2 service Behavioral intercession 4 Structural intercession State 5 Introspection 5 6 Is the introspected state consistent with the initialization state? Oracle 7

46 46 Test Experiments (I) (Service interfaces) interface shorttypeparameters{ short ReturnValue (); void InValue (in short v); void OutValue (out short v); void InOutValue (inout short v); short All ( in short v1, out short v2, inout short v3); }; Reification & Behavioral Intercession Built-in types, Strings, Class types, Structures and Arrays Introspection & Structural Intercession interface shorttypeattributes{ attribute short ReadWriteValue ; attribute readonly short ReadValue ; };

47 47 Test Experiments (II) (object-oriented properties considered) Inheritance: A A C simple B multiple B Encapsulation (methods and attributes): public / protected / private

48 48 Experimental results Reification / Behavioral intercession Method invocations were incorrectly handled using inheritance A foo() B foo() fooid fooid Internal object activity was incorrectly encapsulated int fact(int i){ if (i==0) return 1; return i*fact fact(i-1); } Introspection / Structural intercession Object composition on vs Object references deep copy/restore reference internal object shallow copy/restore external object

49 49 About testing MOPs Step forward for testing reflective systems Reusing mechanisms already tested for testing the remaining ones. Case Study: : feasibility f and effectiveness of the proposed approach Automatic generation of test case input values Guidelines for MOP design Generalizing the approach Multi-level reflective systems Future work Aspect-oriented programming Testing reflection # Reflection for testing

50 50 Conclusion MOPs for FT architectures Language reflection / middleware not reflective CORBA Portable Interceptors Support for FT too limited Unified approach for multi layered open systems Multi-level reflection Validation of the platform Test : augment the confidence FI : failure mode analysis feedback on FT mechanisms