1 ARCHITECTURES FOR TESTING DISTRIBUTED SYSTEMS Andres Ulrich, Hrtmut König Siemens AG, Corporte Technology, ZT SE 1, 81739 München, Germny E-mil: ndres.ulrich@mchp.siemens.de BTU Cottus, Deprtment of Computer Science, PF 101344, 03013 Cottus, Germny E-mil: koenig@informtik.tu-cottus.de Astrct Keywords: Stilizing network infrstructures hs moved the focus of softwre system engineering to the development of distriuted pplictions running on top of the network. The complexity of distriuted systems nd their inherent concurrency pose high requirements on their design nd implementtion. This is lso true for the vlidtion of the systems, in prticulr the test. Compred to protocol testing the test of distriuted systems nd pplictions requires different methods for deriving test cses nd for running the test. In this pper, we discuss rchitectures for testing distriuted, concurrent systems. We suggest three different models: glol tester tht hs totl control over the distriuted system under test (SUT) nd, more interestingly, two types of distriuted testers comprising severl concurrent tester components. The test rchitectures rely on grey-ox testing pproch tht llows to oserve internl interctions of the SUT y the tester. In order to ssure correct ssessment of the test dt collected y the distriuted tester components, the tester hs to mintin correct glol view on the SUT. This cn e chieved either y the use of redundnt points of control nd oservtion or y test coordintion procedures. We outline the fetures of ech pproch nd discuss their enefits nd shortges. Finlly, we descrie possile simplifictions for the rchitectures. Distriuted systems, concurrency, test rchitectures, specifiction-sed testing, concurrent trnsition tour. 1 INTRODUCTION Designing, implementing, nd vlidting complex distriuted systems requires deep understnding of the communiction tking plce nd the
order of communiction messges exchnged etween the individul components to void unwnted ehvior during runtime. This pplies in prticulr to the testing of these systems. Compred to protocol testing the test of distriuted systems nd pplictions requires different methods for deriving test cses nd for running the tests. In [12] we introduced the concept of concurrent trnsition tour n pproch for providing test cses for distriuted systems. The derivtion of concurrent trnsition tours ws represented in [13]. In this pper we continue the pproch with the discussion of possile test rchitectures to run these tests. We ssume tht distriuted system consists of collection of components, ech of them relizing sequentil flow of execution. Ech component hs n interfce tht defines its incoming nd outgoing Messges. The test of distriuted systems usully follows the steps of single component tests, integrtion tests of components, nd finlly the system test. Especilly, the integrtion test is minly ggrvted y the following properties of distriuted pplictions: true concurrency etween components, sence of glol clock, nd messge exchnges etween components tht re unoservle y tester. We ssume tht test rchitecture for distriuted systems consists of the following sic elements: the system under test (SUT), i.e. the executle implementtion of the distriuted softwre system to e tested, the tester tht implements test cse nd lso ssesses the results of test run, points of control nd oservtion (PCOs) etween tester nd SUT, nd possily test coordintion procedures which coordinte the ctions etween different tester components. The pper defines requirements to the test rchitecture tht need to e mtched to support the execution of test runs nd their correct ssessment fterwrds. These requirements ddress the SUT, which must e prepred in suitle wy to perform test run. Requirements in this ctegory re referred commonly to rules of design for testility. Further requirements re suited to design testers tht correctly ssess the results of test run. Note tht our discussion is confined for the testing of functionl spects. We do not consider qulity-of-service nd performnce spects of distriuted systems tht re discussed in [15]. Two principl test rchitectures re presented: glol tester tht oserves nd controls ll distriuted components of SUT in centrl mnner, nd
distriuted tester tht consists of numer of distriuted, concurrent tester components, ech of them collecting only prtil informtion out the execution progress in components of the SUT. Especilly, distriuted tester is of interest since it mkes etter use of system resources nd supports concurrency within the SUT directly. Wheres the glol tester cn e performnce ottleneck during test run. Assuming grey-ox testing pproch nd the right choice of points of control nd oservtion (PCOs) etween tester nd SUT, we show tht lso distriuted tester works correctly nd rings up the expected result. The PCOs must e provided in the implementtion of the SUT to llow the tester to oserve necessry informtion for ssessing test run. A common mens for this purpose is the insertion of softwre proes into the SUT. The pper is orgnized s follows. In Section 2 we introduce some sic notions s well s n exmple which re used in the following discussion. Section 3 descries the proposed test rchitectures. Section 4 discusses possile simplifictions for the ppliction of these rchitectures. The finl remrks summrize the results nd give n outlook on needed further reserch. 2 PRELIMINARIES 2.1 Bsic Nottions We suppose specifiction of distriuted system s prllel composition I = M 1 M k of intercting components. Ech component relizes certin function of the distriuted system, e.g. in the form of client nd/or server. It is descried y sequentil utomton (leled trnsition system, LTS). Components communicte with ech other solely vi interction points. The communiction pttern used is synchronous communiction nd nonlocking send sed on interleving semntics. Trnsmitting messges nd their receipt through interction points re referred to ctions. Definition 1. A leled trnsition system (LTS or mchine for short) M is defined y qudruple (S, A,, s 0 ), where S is finite set of sttes; A is finite set of ctions (the lphet); S A S is trnsition reltion; nd s 0 S is the initil stte. To distinguish the different kinds of communiction, we denote ll inputs nd outputs of the distriuted system implementtion from/to the environment s externl (rective system), nlogously ll inputs nd outputs elonging to the
inter-component communiction s internl. Events ppering only inside module re not considered. Consider the simple system I = A B whose LTSs re given in Figure 1. Under the ssumption tht ctions nd c in ech LTS synchronize, removl of prllel opertor y pplying interleved-sed semntics rules yields the composite mchine C I. d 0 2 A c 1 0 1 B e c C I d 01 11 c 21 e e e 00 10 20 d Figure 1. Exmple system I = A B nd its composite mchine C I. 2.2 Test Suites for Concurrent Systems A conventionl pproch to test suite genertion strts from the composite mchine. The genertion of trnsition tour from the interleving model of the composite mchine hs its limittions since concurrent events re serilized. Due to lck of controllility during testing, this pproch is not fesile. The resulting order of concurrent events in test run could not e predicted. The order of events is, however, essentil to ssess whether n implementtion is correct or not. Possile test cses of our exmple derived from the composite mchine in Figure 1 re the following sequences: σ 1 =..c..d.e., σ 2 =..c..e.d., σ 3 =..c.e..d.. All three sequences cn e eqully employed in test run. They possess the sme fult detection cpility [7]. They differ only in different orders of the interleving ctions, d nd e. However, this order cnnot e predicted in sequentil test run. In [12], we extended the notion of trnsition tour [8] to concurrent trnsition tour (CTT) nd pplied it s test suite for distriuted systems. In the context of distriuted systems, trnsition tour is extended to CTT such tht ll trnsitions in ll modules of the system re visited t lest once on the shortest possile pth through the system. A CTT tkes concurrency mong
ctions of different modules into ccount. It is depicted grphiclly s time event sequence digrm where nodes re events nd the directed rcs define the cuslity reltion etween events. A fesile construction lgorithm of CTT is presented in [13]. Definition 2. A lposet (leled prtilly ordered set) is defined y the qudruple (E, A,, l), where E is set of event nmes; A is set of ction nmes; is prtil order expressing the cuslity informtion etween events, i.e. e f if event e precedes event f in time; l: E A is leling function ssigning ction nmes to events. Ech leled event represents n occurrence of the ction lelling it, with the sme ction possily hving multiple occurrences. A pomset (prtilly ordered multiset) is n isomorphism clss over event renming of n lposet, denoted [E, A,, l]. Definition 3. (CTT) A concurrent trnsition tour of concurrent system I is the lest pomset CTT = [E I, A I,, l] such tht ll ctions of I re covered t lest once in the pomset, i.e. A I for ll ctions in I nd E I is miniml. Figure 2 depicts the CTT of the exmple system I = A B. s t r t c e d e n d Figure 2. A CTT of system I = A B. The CTT is used s test cse to test distriuted system. The CTT ssumes tht lso the internl ctions within the SUT re visile to the tester. Therefore, it supports the grey-ox testing pproch. To gurntee full coverge of ll executle trnsitions of distriuted system, usully set of CTTs is required tht mkes up the test suite for the distriuted system. 2.3 Assumptions on Test Architecture A test rchitecture is defined s prllel, synchronous composition of the system under test (SUT) of the distriuted system I nd its tester. If test suite
comprises severl test cses, seprte tester is uilt for ech test cse. It is required tht the numer of LTSs in the SUT is constnt when the test run tkes plce (sttic system). Furthermore, the structure of the system I, i.e. its composition of n LTSs, is preserved in the implementtion. From the testing perspective, it should e stted wht ctions cn e oserved nd controlled. To void nondeterministic execution of the SUT due to nonoservility, we ssume tht ll internl nd externl ctions re oservle during testing (grey-ox testing pproch). In testing distriuted systems the informtion out ction nmes oserved during test run is not sufficient to ssess conformnce etween specifiction nd SUT. Due to the existence of multi-rendezvous mong components of the SUT, the tester must lso know wht components prticipte in specific multi-rendezvous. Furthermore, the issue of nondeterminism within the SUT requires tht the tester hs the power not only to oserve n ction of the SUT, ut must lso control the occurrence of multi-rendezvous. Thus, the crucil point in testing concurrent systems is to perform deterministic test run. The prolem is ddressed y pplying instnt reply techniques used for deugging concurrent systems [10]. The proposl in [10] ssumes glol controller tht is sked for permission y the components of the SUT efore they re llowed to interct with ech other. To chieve this, control code, clled proes, is dded into the source code of the components efore ech interction invoction. Only fter the tester hs received ll requests to execute certin interction from prticipting components, it grnts this interction. If not or if wrong component sks for permission, n error in the SUT hs occurred. 3 TEST ARCHITECTURES FOR TESTING DISTRIBUTED SYSTEMS 3.1 Glol Testers In this section, we descrie possile test rchitectures tht cn e employed for the test of distriuted systems. We first strt with the simplest model, the glol tester, tht entirely simultes the environment of the STU s sequentil process during test run. The glol tester centrlly collects informtion of the distriuted SUT nd derives the test verdict. A glol tester is implemented s sequentil mchine T G. The tester runs in prllel with the distriuted system oserving nd controlling if necessry ll externl nd internl ctions of the SUT (grey-ox test):
T G SUT After strting the test run the tester executes the events of the test sequence σ step y step. This leds to rendezvous or multi-rendezvous communiction etween the tester nd the SUT. The tester prticiptes in the execution of test event nd records it together with the components of the SUT prticipting in this test event. The glol tester performs n ction sequence derived from the composite mchine of the distriuted system s test cse. This ction sequence contins certin, ssumed interleving order of concurrent events. This ssumed order must e vlidted to exist in the SUT during the test run. However since the interleving sequence consists of concurrent test events with no cusl dependency etween them, the ction sequence ssumed in the glol tester is only oserved y chnce during test run. The glol tester itself is modeled s n cyclic mchine T G = (S T, A CCT, T, S T0 ) with σ +1 sttes. The trnsitions in T G re determined y the sequence of ctions σ = 1. 2.. n used s the test cse: S T0 1 S T1, S T1 2 S T2,, S Tn-1 n S Tn For ssigning the test verdict, verdict lel is ttched to ech stte: pss to the finl stte nd fil to ll other sttes. If the tester reches the finl stte fter correctly executing σ, the test verdict pss is ssigned to the test run. In ll other cses the test run results in fil verdict. The tester my not rech the finl stte if desired component of the SUT is not le to prticipte in certin test event; the test rchitecture including the tester nd the SUT dedlocks in this cse. Figure 3 shows the test rchitecture with glol tester to test the system SUT I = A B. Note gin tht the tester hs ccess to the internl ctions of the SUT. The model of glol tester implementing σ 1 =..c..d.e. s test cse is given in Figure 4. The dvntge of the glol tester pproch is its simplicity. Due to its glol view on the distriuted system under test, it cn register the processed test events immeditely s unique sequence which preserves the correct cusl dependencies etween the ctions of the SUT. A severe drwck of the glol tester is however tht it requires strict control over the execution of concurrent test events. This control might hevily intervene in the originl ehvior of the SUT. The question is whether there re other options to relize test rchitecture tht tkes ttention to the concurrency of ctions in SUT.
SUT I A B d c e T G Figure 3. The test rchitecture with glol tester T G for system SUT I = A B. T G fil fil fil c fil fil pss fil e fil d Figure 4. A glol tester to test the system SUT I = A B. 3.2 Distriuted Testers 3.2.1 Generl Chrcteristics A distriuted tester is chrcterized y the following properties: It consists of severl concurrently operting tester components which process together, ut independently glol test cse (TC). The tester cn e descried in the sme wy s the SUT, e. g. s set of communicting mchines. Ech tester component executes prtil test cse (PTC). A PTC is projection of the glol test cse TC which comprises only those events which cn oserved t the prticulr tester component. The cusl dependencies etween the test events of PTC re determined y the glol TC. Ech tester component oserves suset of the set of ll PCOs. Their selection nd ssignment to tester component is decision tken y the designer of the test rchitecture. The ehvior of the tester components is controlled y Test Coordintion Procedure (TCP).
The generl issue in distriuted tester design is tht the tester my ssign successful verdict to test run lthough the SUT contins fults. This is possile ecuse there is no glol view on the SUT if distriuted tester is used. This prolem cn e tckled y TCP etween the distriuted tester components, either y introducing synchroniztion events into the prtil test cses of tester components or y the use of redundnt PCOs to oserve internl events of the SUT simultneously y severl tester components. For exmple, if component of the SUT sends messge to nother component, one tester component oserves the send event of this communiction, wheres nother tester component oserves the resulting receive event. We cn descrie distriuted tester T D y mens of set of concurrent tester components T D = T D1 T D2 T Dn which re controlled y TCP. Ech tester component processes sequentil prtil test cse in n cyclic mchine tht might e ugmented with synchroniztion events. We further ssume the sme mrkings of fil nd pss to ssign test verdict s discussed for the glol tester. In test run, the distriuted system (T D1 T D2 T Dn ) SUT is executed. The tester components nd the SUT prticipte in the respective test events nd trnsfer from one locl stte to the next. If test event or synchroniztion event of TCP cnnot e executed, the whole test rchitecture dedlocks. Only if ll tester components rech their finl stte, the verdict pss is ssigned to the test run. 3.2.2 Distriuted Testers with Synchroniztion Events A common distriuted test rchitecture tht is often used in testing distriuted telecommuniction systems uses synchroniztion events to implement TCP. We discuss this type of distriuted testers first. We develop distriuted tester 1 T D for the exmple system in Figure 1 nd ssume two tester components: 1 T D1 oserves the ctions nd d of the SUT, while 1 T D2 tests the ctions, c nd e. 1 T D = 1 T D1 1 T D2 Additionlly, the tester components exchnge the synchroniztion event sync. Figure 5 shows this test rchitecture. The derivtion of test cses for the tester components is done s projection of test events oservle y the prticulr tester component from the glol test cse.
SUT I A B d c e 1 T 1 D1 sync T D2 1 T D Figure 5. Test rchitecture with distriuted tester 1 T D nd synchroniztion event sync. Let us ssume the CTT in Figure 2 s the glol test cse for system I. The projected nd linerized prtil test cses for the two tester components re the following: 1 σ 1 =..d for tester component 1 T D1 nd 1 σ 2 =.c.e.. for tester component 1 T D2. These sequences must e supplemented with synchroniztion events. Synchroniztion events re included when the control goes over from one tester component to the other. s t r t c e d e n d Figure 6. The CTT of the exmple system with rrows denoting necessry synchroniztion events within the prtil test sequences. We consider Figure 6. It depicts the glol CTT of our exmple system. The ctions of 1 T D1 re represented y squres o, those of 1 T D2 y dimonds. The grey rrows mrk the chnge from one tester component to the other one. At these points synchroniztion event must e included to gurntee the correct glol view on the SUT. Consequently, the tester components run the following prtil test cses: o
1 σ 1 = sync..sync.sync..d.sync 1 σ 2 =.sync.sync.c.sync.e.sync. Note tht the synchroniztion events re indispensle for ssuring the correct glol view on the SUT. If, for instnce, the first ppernce of ction in component A of the SUT follows ction c, distriuted tester without test coordintion procedure would not detect this fult. Assuming tht tester component is ssigned to ech component of the SUT, the distriuted tester cn fully exploit the concurrency mong test events. For exmple, the totl order of the interleving ctions, d nd e is not importnt for ssessing test run. Any test run is correct if only the cusl dependency efore d is ssured. Action e cn interleve t ny time instnt within the constrints of the sync events. Thus, the distriuted tester does not need to control the execution of the test cse entirely. Ech tester component cn run independently its prtil test cse. Only coordintion etween the tester components is necessry to gurntee the correct glol view on the SUT. 3.2.3 Distriuted Testers with Redundnt Oservtion of Internl Events The insertion of synchroniztion events is not the only possiility to relize coordintion mong tester components. Another nd more elegnt option is the redundnt oservtion of internl ctions of the SUT y severl tester components. This type of distriuted test rchitecture is discussed gin using the exmple from Section 2. It consists of two tester components 2 T D = 2 T D1 2 T D2. Ech tester component controls nd oserves ll interctions of single component of the SUT. Thus, the tester represents n inverted imge of the SUT. Figure 7 shows the test rchitecture of tester 2 T D. Prtil test cses for the tester components re derived in the sme mnner s lredy descried for the first type of distriuted testers in Section 3.2.2 y projecting nd linerizing the glol test cse of CTT. We lso ssume the sme mrking of sttes in the tester components with fil nd pss s discussed for the glol tester. Using the CTT in Figure 2 s the sis for the derivtion of prtil test cses, the tester components 2 T D1 nd 2 T D2 need to implement the following prtil test cses: 2 σ 1 =..c..d. for tester component 2 T D1 nd 2 σ 2 =.c.e. for tester component 2 T D2.
SUT I A B d c e 2 T D1 2 T D2 2 T D Figure 7. Test rchitecture with the distriuted tester 2 T D nd redundnt oservtion of test events. Since oth tester components oserve the internl ctions of the SUT, they re le to coordinte ech other to mintin the correct glol view on the system. In cse of messge pssing, prcticl solution to implement the redundnt oservtion of internl communiction is the oservtion of the send event of messge y one tester component nd the oservtion of the relted receive event y nother one. The cusl dependency etween the send nd the receive event for the sme communiction ction (messge pssing) cn then e reconstructed. In test run the distriuted system (T D1 T D2 T Dn ) SUT is executed. The tester components nd the SUT prticipte in the respective test events. A multi-rendezvous etween the SUT components nd the tester components tkes plce when internl ctions of the SUT re executed s test events. In this cse, the prticipting tester components synchronize to preserve the glol view on the SUT. If test event cnnot e executed, the tester components ssocited to this test event dedlock. Only if ll tester components rech their finl stte, the verdict pss is ssigned to the test run. Note lso here tht concurrency within the SUT is supported to lrge extend if tester component is provided for ech component of the SUT. The dvntge of this second distriuted test rchitecture is tht no dditionl synchroniztion events re needed to coordinte the tester components.
4 SIMPLIFICATIONS OF THE PROPOSED TEST ARCHITECTURES The test rchitectures presented ove contin severl restrictive elements. For certin pplictions, the following test ssumptions my e too restrictive: the control nd oservtion of ll ctions of the SUT including the internl ctions, nd inclusion of dditionl synchroniztion events. In the following discussion we shortly sketch some conditions which permit simplifictions of the test rchitecture. More detils re given in [14]. 4.1 Oservtion of Internl Communiction The complete oservtion of the internl communiction (grey-ox test) ws introduced to force deterministic test run. By oserving the internl communiction of the SUT y mens of controller ccording to [10] the SUT is forced to follow certin execution order defined y the CTT. To (prtilly) omit the internl oservtion, it must e proved whether the ehvior of the SUT does not led to nondeterministic ehvior which cnnot e detected y the tester. This requires tht we need to tke into considertion the following types of nondeterminism * : nondeterminism within component, i.e., it exists the possiility of two or more outputs in the sme locl stte, nd rce conditions, i.e., it exists glol stte with t lest two inputs to the sme component enled. If the nlysis of the specifiction of the SUT shows tht these types of nondeterminism do not pper, dditionl mens to force deterministic test run cn e omitted. This pproch requires however tht the components of the SUT re correctly implemented. To ssure this step-wise test method s proposed in [4] cn e pplied. * Another kind of nondeterminism is interleving, since the selection mong two concurrent ctions is ritrry nd unpredictle. We do not consider interleving s nondeterminism here, ecuse we llow concurrent ctions in test run to e crried out if distriuted tester is used.
4.2 Renuncition of Test Coordintion Procedures In cses in which the numer of dditionl synchroniztion events tht re needed to implement the distriuted tester of Section 3.2.2 is very high or in cses in which the redundnt oservtion of internl ctions of the SUT s required for the distriuted tester of Section 3.2.3 is not possile, it might e desired to renounce the test coordintion procedures. To synchronize the then synchronously cting tester components, the following possiilities exist: Simultion of glol clock y mens of synchroniztion protocol There exist severl such protocols in literture. A well-known one is the Network Time Protocol (NTP) [5]. It supports synchroniztion of few milliseconds divergence for locl re networks nd some 10 milliseconds for wide re networks. This precision is scrcely cceptle for mny prcticl pplictions s the following mesurements of the trnsmission time of RPC messges show. RPC (Remote Procedure Cll) is network service for the trnsprent execution of procedures on remote mchines. It is the sic service used y the middlewre pltform CORBA. The trnsmission time of n out 8 kiloyte RPC messge over n Ethernet network is 7,2 msec. For 100 Byte messge, the time is 0,29 msec only. In n ATM network with 155 Mps trnsfer rte, these times shorten to 0,5 nd 0,02 msec, respectively. These mesurements indicte tht within the precision limits of NTP (1 msec in the est cse), severl messges cn e sent over n Ethernet such tht the cusl dependencies etween messges cnnot e correctly estlished in test run. For ATM nd other high speed networks, the sitution is even worse. Use of logicl clocks in the test rchitecture Logicl clocks [2] re very relile nd roust mens to synchronize ctions in distriuted system. It requires, however, tht logicl clocks re ville in ll components of the distriuted system, i.e. lso in the SUT. Tht mens, the softwre of the SUT must e ugmented such tht logicl clock vlues re piggycked for ech messge exchnged etween SUT nd tester nd etween internl SUT components. Emultion of synchronous execution y mens of synchronizer In our current discussion we ssumed n synchronous execution etween the components of the SUT. A synchronized execution cn e forced y introducing synchronizer tht determines the time for the execution of communiction event [1]. However, synchronous execution of components in distriuted system cnnot e ssumed for most implementtions. Moreover,
the introduction of synchronizer for the purpose of testing only would chnge the implementtion enourmously. An pproch tht works similr to synchronizer is discussed in [3]. Genertion of synchronizle test suites Synchronizle test suites which cn e seprtely executed on different tester components without synchroniztion events re n ttrctive solution for this prolem. Existing pproches [6], [11] run short ecuse they cn only e pplied if glol composite mchine for the distriuted SUT is ville. Approches tht support communicting mchines s the description model for the SUT do not exist up to now. 5 FINAL REMARKS In this pper we hve suggested different rchitectures for the test of distriuted systems. We hve discussed rules for the positioning of PCOs s well s mens to ensure deterministic test run. Two principle test rchitectures hve een considered: glol nd distriuted tester. The glol tester is sequentil component tht oserves ll ctions of the SUT t its PCOs, wheres the distriuted tester consists of severl concurrent tester components tht oserve prtil ehvior of the SUT only. These tester components must synchronize ech other y mens of test coordintion procedure to ssure correct nd consistent glol view on the SUT. The presented pproches se on synchronous communiction prdigm etween the components of the SUT nd the test rchitecture. The ssumption of synchronous communiction is required for two resons. First it restricts the stte spce of the distriuted systems to finite set, nd secondly it llows to preserve the correct cusl dependencies etween the test events. Using n synchronous communiction prdigm, these properties would not hold nymore. The stte spce would e infinite nd the cusl dependencies etween test events could not e correctly ssessed. Nevertheless, it is necessry lso to pursue reserch within this direction, ecuse synchronous communiction is common in prctice. REFERENCES [1] C. T. Chou, I. Cidon, I. Gopl, S. Zks: Synchronizing synchronous ounded-dely networks; IEEE Trnsctions on Communictions, vol. 38, no. 2 (Fe. 1990); pp. 144 147.
[2] C. J. Fidge: Logicl Time in Distriuted Computing Systems; IEEE Computer, vol. 24, no. 8 (August 1991); pp. 28-33. [3] C. Jrd, T. Jeron, H. Khlouche, C. Viho: Towrds Automtic Distriution of Testers for Distriuted Conformnce Testing; IFIP Joint Int l Conference on Forml Description Techniques, nd Protocol Specifiction, Testing, nd Verifiction (FORTE/PSTV 98); Pris, Frnce; 1998. [4] H. König, A. Ulrich, M. Heiner: Design for Testlility: A Step-wise Approch to Protocol Testing; Proceedings of the IFIP 10th Int l Workshop on Testing of Communicting Systems (IWTCS'97), Seoul, Kore; 1997; pp. 125-140. [5] D. L. Mills: Precision Synchroniztion of Computer Network Clocks; ACM Computer Communiction Review, vol. 24, no.2 (April 1994; pp. 28-42. [6] G. Luo, R. Dssouli, G. v. Bochmnn, P. Venktrm, A. Ghedmsi: Test genertion with respect to Distriuted Interfces; Computer Stndrds & Interfces, vol. 16, no. 2 (June 1994); pp. 119 132. [7] A. Petrenko, A. Ulrich, V. Chpenko: Using prtil-orders for detecting fults in concurrent systems; Proceedings of the IFIP 11th Int l Workshop on Testing of Communicting Systems (IWTCS'98), Russi, 1998. [8] Sidhu, D. P.; Leung, T. K.: Forml methods for protocol testing: detiled study; IEEE Trns. on Softwre Eng. 15 (1989) 4, 413 426. [9] K. C. Ti, R. H. Crver: Testing of distriuted progrms; in A. Zomy (ed.): Hndook of Prllel nd Distriuted Computing; McGrw Hill; 1995; pp. 956-979. [10] K. C. Ti, R. H. Crver, E. E. Oid: Deugging concurrent Ad progrms y deterministic execution; IEEE Trns. on Softwre Eng., vol. 17, no. 1 (Jn. 1991); pp. 45-63. [11] K. C. Ti, Y. C. Young: Port-synchronizle test sequences for communiction protocols; 8th Int l Workshop on Protocol Test Systems (IWPTS 95); Pris, Frnce; 1995; pp. 379 394. [12] A. Ulrich, S. T. Chnson: An pproch to testing distriuted softwre systems; 15th PSTV 1995; Wrsw, Polnd; pp. 107-122; 1995. [13] A. Ulrich, H. König: Specifiction-sed testing of concurrent systems; IFIP Joint Int l Conference on Forml Description Techniques, nd Protocol Specifiction, Testing, nd Verifiction (FORTE/PSTV 97); Osk, Jpn; Nov. 18-21, 1997. [14] A. Ulrich: Testfllleitung und Testrelisierung in verteilten Systmen; Disserttion (in Germn), Mgdeurg University; Shker Verlg, 1998. [15] T. Wlter, I. Schieferdecker, J. Growski: Test rchitectures for distriuted systems: stte of the rt nd eyond; 11th Int l Workshop on Testing of Communicting Systems (IWTCS'98), Russi, 1998.