χ=5 virtual time state LVT entirely saved state partially saved state χ=5 ν=2 virtual time state LVT entirely saved partially saved unsaved state

Transcription

1 ROLLBACK-BASED PARALLEL DISCRETE EVENT SIMULATION BY USING HYBRID STATE SAVING Francesco Quaglia Dipartimento di Informatica e Sistemistica, Universita di Roma "La Sapienza" Via Salaria 113, Roma, Italy, quaglia@dis.uniroma1.it Vittorio Cortellessa Dipartimento di Informatica S&P, CERTIA Research Center, Universita di Roma Torvergata Via della Ricerca Scientica, Roma, Italy, cortelle@info.utovrm.it KEYWORDS Parallel discrete event simulation, hybrid saving, rollback-recovery mechanisms, performance evaluation. ABSTRACT Optimistically synchronized parallel discrete event simulators must sometimes undo, by rolling back parts of the system, the erroneous over optimistic computation deriving from the decentralized management of the event list. For this reason, an essential part of these simulators is the saving mechanism. Three saving mechanisms have been proposed in literature: copy, periodic and incremental saving. In this paper we introduce a new saving technique, that will be referred to as hybrid, which mixes the advantages of previous approaches. We also present experimental results obtained in a simulation environment which adopts hybrid saving; such results quantify the benets, in terms of reduced simulation execution time, achievable by using our technique. 1 INTRODUCTION In parallel discrete event simulation the simulation program is partitioned into a number of logical processes (LPs), which model the behavior of dierent parts of the simulated system (Fujimoto 1990). The interaction between LPs is realized by message exchange; messages are stamped with a virtual time value (timestamp) that indicates when, in virtual time, the receiving LP must process them. The processing of a message determines the execution of an event which moves an LP from one to another and eventually produces some new messages that can This work is partially supported by Scientic Cooperation Network of the European Community \OLOS" under contract No. ERB4050PL and by the University of Roma at "Tor Vergata" CERTIA Research Center Project on Multimedia and Collaborative Technology, the MURST Projects on Performability in Software Engineering and Performance of Client-Server Systems, and the CNR Project on Performance and Reliability Engineering of Distributed Databases be addressed to any LP. Each LP has its own simulation clock (local virtual time), and an event list in which incoming messages are enqueued. Optimistic approach allows each LP to execute asynchronously. However, in order to ensure correct simulation results, certain causality constraints must be met. Specically, each LP must process received messages in non decreasing timestamp order. The most common optimistic method is referred to as Time Warp (Jeerson 1985). Under Time Warp no LP undergoes constraints in order to process a message, thus local simulation clocks may diverge. For this reason an LP may receive a message with a timestamp that is smaller than its current local virtual time (straggler message), breaking the causality constraints between events. In this case, the over-optimistic portion of the simulation is undone by rolling back the LP to a previous. From this the LP resumes its computation. The implementation of the rollback procedure requires the ability of restoring, at run time, a past of an LP. To this purpose, a simple mechanism, often called copy saving, has been proposed (Jefferson and Sowizral 1982; Jeerson 1985). It consists of copying into a queue the entire of an LP each time it executes an event. According to this solution the to be restored (due to rollback) is always available, but the saving overhead usually reaches unacceptable levels. Commonly, rollbacks are unfrequent compared to ordinary event execution, so, given that an LP can be regenerated from an earlier one by simply reexecuting some intermediate events, the periodic saving technique has been introduced for reducing the overhead due to copy saving (Bellenot 1992; Fujimoto 1990). According to this solution only a subset of LP s are saved, and the number of events executed between successive saving operations is known as checkpoint interval of an LP. For simulation models in which at each event execution only a fraction of the variables of the LP are modied, a third saving technique has been proposed, namely incremental saving (Steinman 1993; Unger et al. 1993). This solution consists of saving, at each event execution, only the inverse of

2 the changes of an LP, so that a past can be regenerated by applying, one at a time, the saved changes in backward order. In this paper we propose a new saving technique, that will be referred to as hybrid saving, which mixes the benecial eects of both periodic and incremental saving. According to this technique, an LP periodically saves its, but it has also the ability of saving the inverse of the incremental changes of its occurring in a checkpoint interval. In this way an LP is able to regenerate a past either by starting from an earlier saved and re-executing some intermediate events, or by applying to a later saved its previous inverse changes. We compare our technique to the above mentioned ones which, as it will be shown, can be obtained as a particular case of hybrid saving. We both design and propose an implementation of the hybrid saving protocol and we show experimental results which quantify the advantages, in terms of reduced simulation execution time, achievable by using such protocol compared to preexisting proposals. The paper is organized as follows: in Section 2 we describe actually used saving techniques, in Section 3 we introduce the hybrid saving protocol and we present a simulation environment adopting hybrid saving, experimental results which quantify the performance improvements achievable by using our technique are shown in Section 4, short conclusions constitute Section 5. 2 BACKGROUND In this section we give some details of actually used saving mechanisms in Time Warp simulators. 2.1 Copy and Periodic State Saving When copy saving is adopted, the of an LP is saved into a queue before a new event is executed. Usually, the saved is also marked with the local virtual time () of the LP (that is the timestamp of the last processed message). In this way all the s passed through by an LP are available and the rolling back of the LP to a virtual time T is realized by simply restoring the most recent marked with smaller than T. Upon rolling back, the LP cancels from the queue all the recorded s with larger than or equal to T. This technique adds to each event execution a checkpointing overhead, which is quantied by the sum of the time required to allocate a buer and the time required to copy the current into the buer. One approach for reducing such overhead is to perform saving every event executions ( being the checkpoint interval of an LP). This solution, known as periodic saving, has both benecial and detrimental eects on the LP execution time. It reduces the number of CPU cycles spent in saving operations, but upon rolling back, the required may not be in the queue. In latter case, such must be recomputed by reprocessing input messages (thus adding a time penalty). An LP that is recomputing a missing is said to be in a coasting forward phase (Fujimoto 1990). Both analytical and experimental studies have been carried out (Lin et al. 1993; Palaniswamy and Wilsey 1993a; Preiss et al. 1994) to point out the relation between periodic saving and the simulation execution time of an LP; several techniques for allowing an LP to dynamically recalculate the value for its checkpoint interval have also been proposed (Fleischmann and Wilsey 1995; Palaniswamy and Wilsey 1993b; Quaglia and Auriche 1997; Ronngren and Ayani 1994), in order to reduce the simulation execution time compared to the one obtained with static periodic sate saving. 2.2 Incremental State Saving Many challenging simulations (for example simulations of large communication systems) are characterized by LPs with large event execution time and with very large (up to hundreds of kilobytes), where only fractions of the are updated at each event execution. In these simulations it may be very inef- cient both saving copies of the complete, and regenerating a past by starting from an earlier one (through reprocessing intermediate events). In such simulations it is often protable to use incremental saving, in which only the inverse of the changes are saved at each event execution. Thus the saving mechanism builds a chain of inverse of the changes that, in case of rollback, allows the reconstruction of a past by starting from the current of the LP and applying the saved changes in backward order till the required is obtained. This solution successfully reduces the saving and reconstruction overhead when both the number of inverse changes saved at each event execution and the rollback length are minimal (Palaniswamy and Wilsey 1993a). 3 HYBRID STATE SAVING As shown in the previous section, an LP running in a Time Warp simulation must be able to regenerate whichever past, in order to ensure the correctness of the simulation rollback. It can be achieved either by starting from an older saved and reprocessing intermediate events or by backward applying saved incremental changes to the current. The mixing of such mechanisms consists in allowing an LP to start from whichever saved and to regenerate the required either by reprocessing

3 Τ1 Τ entirely saved χ=5 Τ2 partially saved virtual time Figure 1: an example of hybrid saving Τ1 unsaved Τ χ=5 entirely saved ν=2 Τ2 partially saved virtual time Figure 2: hybrid saving with = 5 and = 2 events (forward regeneration) or by backward applying saved incremental changes (backward regeneration). This mixed approach can be achieved by periodically saving the LP, and by saving the inverse of the changes in each checkpoint interval. In Figure 1 we show a portion of the evolution of an LP (i.e., a portion of the s passed through by the LP) which records whole its each ve event executions ( = 5), and saves all the inverse of the changes that occur in a checkpoint interval. In the rest of the paper we denote as partially saved the copy of the variables which are going to be modied by the next event execution, while we denote as entirely saved the copy of the whole LP. The LP whose evolution is shown in Figure 1 is able to regenerate the with LV T = T either by starting from the with LV T = T 1 and reprocessing two events, or by starting from the with LV T = T 2 and applying three inverse changes to the variables. However, by saving all the inverse of the incremental changes that occur in a checkpoint interval, only the length of the LP checkpoint interval can be tuned for reducing the total overhead due to the rollback-recovery mechanism. A generalization of the structure shown in Figure 1 is easily obtained by allowing an LP to save inverse changes of its starting from whichever in the checkpoint interval. In Figure 2 we show a case in which the LP saves its entire after ve event executions, while it partially saves two s in a checkpoint interval. In this case two parameters can be tuned: the checkpoint interval and the number of partially saved in each checkpoint interval. On the other hand, in this case the LP is not able to regenerate the with LV T = T by starting from the with LV T = T 2 and applying inverse changes, but it must necessarily regenerate that by starting from the with LV T = T 1 and by executing a coasting forward phase. Given that, at most,?1 s are partially saved in each checkpoint interval we get: 0 = < 1. As last consideration, copy, periodic and incremental saving can be generated starting from hybrid saving and by assigning appropriate values to the parameters and. In particular: - when = 1 (thus is necessarily 0) we get copy saving; - when > 1 and = 0 we get periodic saving; - when =? 1 and goes to innity we get incremental saving. We have implemented hybrid saving on the distributed simulation platform SIMCOR (Ciciani and Angelaccio 1994). This platform has been realized for carrying out parallel optimistically synchronized simulations on the hypercubical machine ipsc/2 (the platform's software is realized in C). In SIMCOR, LPs are statically assigned to processors; the aggressive approach (Gafni 1985) is adopted in the cancellation phase (i.e., antimessages are sent as soon as an LP rolls back). A single scheduler (Time Warp kernel) runs on each processor and manages the local event list by scheduling local LPs according to the STF (Shortest-Timestamp-First) algorithm (Jefferson 1985). The scheduler also manages the queue of each LP. In this way, both the scheduling and the saving mechanisms are transparent to the LPs. Inter-process communication is realized by using routines which are supported by the NX/2 distributed operating system running on the ipsc/2 machine. NX/2 also supports synchronization primitives which are used for computing GVT and executing fossil collection of obsolete totally/partially saved s and buered messages. In SIMCOR each message is stamped with its own type: for example, in queueing networks, the NEW- CUSTOMER type schedules the arrival of a new customer, while the END-SERVICE type schedules the end of the service for a customer. 3.1 The State Saving Protocol On each processor, the scheduler manages two arrays: the i-th entry of one array records the checkpoint interval of the i-th LP; the i-th entry of the other array records the number of the inverse of the changes that must be saved in a checkpoint interval for the i-th LP. In addition, each LP is endowed with a counter denoting the number of processed events from the last entirely saving operation of the LP. Each time the scheduler extracts a new event for a local LP (and before the LP executes such event), it checks (by using the corresponding counter) whether

4 pointer to the previous buffer of the LP F new buffer pointer to the new buffer Figure 3: structure of a buer the whole of the LP must be saved (i.e., if events have been processed from the last entirely saving ), or just the inverse of the changes must be recorded (i.e., if more than?? 1 events have been processed from the last entirely saving ), or no saving operation must be executed. In the rst case, the scheduler dynamically allocates a new buer in the LP queue and copies the current process into the buer. The buer is also marked with the of the scheduled LP and with a ag F which indicates whether the buer contains a copy of the entire or not (the structure of the buer is shown in Figure 3). In the second case, all updates to variables are identied and a new buer is allocated and linked to the queue only for back up copies of the variables which will be modied by the event execution. In SIMCOR, the identication of the variables that will be modied results quite simple because each message is stamped with its own type; the scheduler, by checking the type of the extracted message, identies the portion of code that will be executed by the LP, thus the corresponding part of the that will be modied. All the buers (containing either an entire or parts of the ) are linked by back pointers. 3.2 The State Regeneration Protocol By adopting hybrid saving a process may regenerate a past either by coasting forward some intermediate events, or by backward applying to a saved its previous inverse changes. Although the rst choice is always feasible, the second one works only if the required past falls in the range covered by the inverse changes saved in a checkpoint interval (this problem has already been shown above). In SIMCOR the protocol for regenerating a past is deterministic and works as follows: if the required is out of the ones covered by starting from a saved and backward applying inverse changes, normal coasting forward is used; otherwise, the is regenerated by backward reconstruction. Furthermore, in case of rollback, the regeneration is executed in atomic fashion (message/antimessage preemption is discarded). If the LP must rollback to the virtual time T, the scheduler searches, into the queue, the couple (S',S") of successive entirely saved such that LV T S 0 < T < LV T S 00 (note that the S" may be not yet saved into the queue; in this case the current LP is considered instead of S"). Then the oldest buer, if any, containing inverse of the changes between S' and S" is searched (S ), and its is compared with T. If T < LV T S then a coasting forward phase is required; the scheduler restores the S' in the LP and forces it to replay some already processed messages. Otherwise the scheduler starts to apply inverse incremental changes to the S" by backward running through the LP queue. Such procedure stops when the rst buer marked with LV T < T is found. The obtained is then restored in the LP. After the LP rolls back, all the buers with larger than T are released. 4 PERFORMANCE EVALUA- TION In this section we propose several experiments to show the benets, in terms of reduced execution time, that can be achieved by using hybrid saving. We compare the simulation execution time spent when hybrid saving is adopted to the one spent by using periodic saving. We propose simulation results of a stochastic queueing model, whose topology is a fully connected net with 32 service centers, where a constant customer population circulates among the centers (three customers in each center at the simulation starting). The timestamp increments are taken from an exponential distribution with mean 1 unit time, and customers are equally likely to be forwarded to any other center. We denote as s the time to save the entire copy of the LP and with e the average time to execute one event (excluding the time for sending the eventually produced messages). In the simulated model, only an average portion of 1=3 of the is modied by the execution of an event, so we approximatively have a s =3 average time spent for saving the inverse of a change (both the saving of the entire and the saving of a portion of the need the dynamic allocation of the corresponding buer, so the time for saving a fraction of the is not exactly proportional to the fraction of the to be saved). We focused our attention on the execution time required for committed events. The proposed set of experiments has been realized by adopting two dierent values of the ratio s = e : s = e = 2 and s = e = 1=2 (the value of the ratio s = e is modied, as in (Preiss et al. 1994), by introducing a variable delay loop into the event execution routine). In this way we can point out an idea of the performance of hybrid saving either when the saving cost dominates or when the event execution cost dominates. We studied the execution time of both periodic saving (referred to as P SS) and hybrid saving (referred to as HSS) while varying the checkpoint interval of the LPs. In the case of HSS, we plot two curves: HSS 1, where = b=2c, and HSS 2, where =? 1.

5 In Figure 4 ( s = e = 2) and in Figure 5 ( s = e = 1=2), the execution times obtained with P SS, HSS 1 and HSS 2 are shown vs. the length of the checkpoint interval of the LPs. The execution time results as the average of 20 runs, and the measures in dierent runs were within 5% of each other of some experiments carried out in a simulation environment supporting hybrid saving. Future work could be focused on both dening the convenient application domain of hybrid saving (e.g., while varying the portion of the modied by the execution of one event), and building algorithms for the dynamic selection of the protocol parameters ( and ). execution time (sec) PSS HSS1 HSS checkpoint interval of the LPs Figure 4: execution time when s = e = 2 The results show that, for this simulation model, HSS improves performances especially for large values of the checkpoint interval. This is because, when is large, P SS suers from an high overhead due to the coasting forward, while HSS reduces such overhead especially for large values of (in fact, the plots show that HSS 2 performs better than HSS 1 ). This phenomenon is clearly evident when considering large event execution time, so the gain achievable by using hybrid saving is larger in the case of s = e = 1=2. In conclusion, the results point out that when grows, the advantages introduced by the reconstruction mechanism of HSS, makes the overhead due to rollback-recovery ever smaller than the one of P SS. 5 CONCLUSIONS In this paper we have introduced a new saving protocol (namely hybrid) for Time Warp simulators. This protocol merges the advantages of the most commonly adopted saving techniques (periodic and incremental). Hybrid saving gains over other techniques by reducing the overhead time due to the rollback-recovery mechanism. The performance improvements achievable by adopting our approach are quantied by the results execution time (sec) PSS HSS1 HSS checkpoint interval of the LPs Figure 5: execution time when s = e = 1=2 REFERENCES Bellenot, S "State skipping performance with the Time Warp operating system", In Proceedings of 1992 SCS Workshop on Parallel and Distributed Simulation (Newport Beach, California, January 20-22). Society for Computer Simulation, Ciciani, B. and M. Angelaccio "An interface to develop Time-Warp based parallel simulations". In Proceedings of 1994 Massively Parallel Processing Conference (Delft, Holland, June 21-23), Elsevier Science, Fleischmann, J. and P.A. Wilsey "Comparative analysis of periodic saving techniques in Time Warp simulators". In Proceedings of 1995 SCS Workshop on Parallel and Distributed Simulation (Lake Placid, New York, June 14-16). Society for Computer Simulation, Fujimoto, R.M "Parallel discrete event simulation", Communications of ACM 33, no.10 (October): Gafni, A "Space management and cancellation mechanisms for Time Warp", Tech. Rep. TR University of Southern California, Los Angeles, California. Jeerson, D. and H. Sowizral "Fast concurrent simulation using the Time Warp mechanism; part I: local control", Tech. Rep. N1906AF. RAND Corporation, (December). Jeerson, D "Virtual time", ACM Trans. on Programming Languages and Systems 7, no.3 (July): Lin, Y.B.; B.R. Preiss; W.M. Loucks and E.D. Lazowska "Selecting the checkpoint interval in Time Warp simulation". In Proceedings of 1993 SCS Workshop on Parallel and Distributed Simulation (San Diego, California, May 17-19). Society for Computer Simulation, Palaniswamy, A.C. and P.A. Wilsey. 1993a. "An analytical comparison of periodic checkpointing and incremental saving". In Proceedings of 1993 SCS Workshop on Parallel and Distributed Simulation (San Diego, California, May 17-19). Society for Computer Simulation, Palaniswamy, A.C. and P.A. Wilsey. 1993b. "Adaptive checkpoint intervals in an optimistically synchronized parallel digital system simulator". In Proceedings of IFIP TC/WG10.5 Int. Conf. on Very Large Scale Integration (September) Preiss, B.R.; W.M. Loucks and D. MacIntyre "Eect of the checkpoint interval on time and space in Time Warp". ACM Transactions on Modeling and Computer Simulation 4, no.3 (July): pp Quaglia, F. and L.R.G. Auriche "A new technique for adaptive checkpointing in Time Warp". In Proceedings of 1997 SCS European Simulation Multiconference (Istanbul, Turkey, June 1-4). Society for Computer Simulation, Ronngren, R. and R. Ayani "Adaptive checkpointing in Time Warp". In Proceedings of 1994 SCS Workshop on Parallel and Distributed Simulation (Edinburgh, Scotland, July 6-8). Society for Computer Simulation, Steinman, J "Incremental saving in SPEEDS using C plus plus". In Proceedings of 1993 Winter Simulation Conference (Los Angeles, California, December). Society for Computer Simulation, Unger, B.W.; J.G. Cleary; A. Covington and D. West "External management system for optimistic parallel simulation". In Proceedings of 1993 Winter Simulation Conference (Los Angeles, California, December). Society for Computer Simulation,