A Cost Model for Distributed Shared Memory. Using Competitive Update. Jai-Hoon Kim Nitin H. Vaidya. Department of Computer Science

Transcription

1 A Cost Model for Distributed Sared Memory Using Competitive Update Jai-Hoon Kim Nitin H. Vaidya Department of Computer Science Texas A&M University College Station, Texas, , USA Abstract Tis paper presents a new \cost" analysis model for distributed sared memory (DSM) using competitive update protocol. Te cost metric of interest ere is te overead of message passing necessary to implement DSM. Tis approac is based on segment model proposed previously [1, 11]. Input parameter for te cost analysis model proposed ere is te probability density function of te number of remote updates in a segment. Tis distribution can quite accurately caracterize many applications. Te proposed model is validated by comparing analytical results obtained using te model to experimental results. Te competitive update protocol for sared memory is dened using a parameter called \update limit" (or tresold). Using te proposed model, we compute te optimal update limit for competitive update protocol. 1 Introduction Software distributed sared memory (DSM) systems ave many advantages over message passing systems [12, 15]. Since DSM provides a user a simple sared memory abstraction, te user does not ave to be concerned wit data movement between osts. (Users can use te DSM as if te sared memory is available locally.) Many applications programmed for a multiprocessor system wit sared memory can be executed on a software DSM system witout significant modications. Software DSM is often implemented on a workstation cluster. Software DSMs typically maintain consistency at te granularity of a page. Tis allows software DSM implementations to use paging ardware typically available in modern workstations. Many dierent protocols ave been proposed for maintaining consistency. Typically, DSMs are based on variations of write- Tis work is supported in part by te National Science Foundation under grant MIP invalidate and/or write-update protocols. A writeinvalidate protocol is similar to read-replication described in [15]. Page replication is allowed for read operation. However, only single copy is allowed for write operation. A write-update protocol is te same as te full-replication described in [15]. Page replication is allowed for bot read and write operations. Te competitive update protocol [7, 6] is dened using a parameter called \update limit" or \tresold". Wen using te competitive update protocol wit limit L, a node A invalidates te local copy of a page P if and wen (L + 1)-t update to te page by oter nodes occurs since te previous access of page P by node A. Te traditional update protocol can be obtained by coosing L = 1. Te protocol obtained wen L = is similar to te traditional invalidate protocol. Tus, selecting appropriate limit L is important in te competitive update protocol. Tis paper presents a new cost analysis model for te competitive update protocol. Tis model is based on probability density function (pdf) for number of updates in a \segment" [1, 11] { a segment for page P is dened as te sequence of remote updates to page P between two consecutive local access of page P by a node. More accurately, a new segment begins wit te rst local access by a node following an update to te page by anoter node. Segments are dened from te point of view of eac node. Terefore, for te same page, dierent nodes may observe dierent segments. Figure 1 illustrates segments for page P observed at node A wit an example: Segment 1 for page P at node A starts at time 1 wen node A reads page P. Copy of page P on node A is ten updated by nodes B, C, and D. After tat, node A starts segment 2 by a local access at time 6. Similarly, node A starts segment 3 by local access at time 11 following remote updates by nodes B and C at time 9 and 1, respectively.

2 remote writes by oter nodes B write C write D write B write C write B write Page P time local access A read A write A read A read by node A A write A write A read A write segments Segment 1 Segment 2 Segment 3 Figure 1: Segments If competitive update protocol wit limit L = 1 is being used, ten page P at node A (refer Figure 1) will be invalidated wen te update by node C occurs in segment 1. Once te page is invalidated, ideally, subsequent updates (for instance, by node D) to page P will not be sent to node A, as it does not ave a valid copy of P anymore. In tis case, wen node A accesses page P at te beginning of segment 2, a page fault will occur (as node A's copy of page P is invalidated). Wen te page fault occurs, node A will send a request for page P to some oter node. Tis request may be forwarded a few times before it reaces a node tat as page P. Tis node will send page P to node A, and node A will ten send an acknowledgement for page P. 2 Related Work Many cost analysis models are presented by oter researcers. Most existing models use read/write ratio and/or memory access fault ratio as input parameters [3, 2, 8, 15, 16, 14]. References [3, 2] present modeling to predict te number of cace misses. Stumm and Zou [15] compare te performance of four basic protocols. Tey compute te average memory access cost based on parameters: te cost of sending packet, te cost of sending page, te number of nodes, read/write ratio, and probability of fault. Srbljic et al. [14] present similar performance analysis based on data access type: single/multiple-reader single/multiple-writer. Kessler and Livny [8] compare te performance DSM algoritms by varying a syntetic memory access pattern caracterized by several parameters. Brorsson and Stenstrom [2] classify sared memory access pattern as read-only/read-write exclusive/sared-by-few/sared-by-many. Tey caracterize producer-consumer and migratory access pattern using tis classication, and compute cace miss ratio. Tey also apply similar metod to stationary access pattern [3]. Eggers [5] presents a write-run model to predict te cace coerency overead for te bus based multiprocessor system. write-run is a sequence of local writes between two consecutive remote accesses. Tis model measures te overead by te number of sequences of single processor writes to an individual sared address, and te lengt of tese sequences. Anderson and Karlin [1] use te write-run model to design adaptive sceme for bus-based systems. (Our model applies to distributed systems.) Raynaud et al. [13] present an update distance model for adaptive update protocol, wic is similar to te segment model. In teir approac, a directory records te update patterns observed and ten uses tem to selectively send updates and invalidations to processors. Selecting appropriate update limit L is important for te competitive update protocol. However, no study as been done for analytically determining appropriate update limit. Previous approaces include selecting update limit by guesswork, or experimental evaluation of te application for many dierent update limits. [6] presents simulation results by varying te update limit for te competitive update protocol. Tey conclude tat te best update limit is dicult or impossible to predict. 3 Cost Analysis Te cost of a message of size m is denoted as c(m). For instance, c(m) may be 1 { tis means tat te cost metric simply counts te number of messages. Anoter possibility is c(m) = m, wic would mean tat te total amount of data sent by messages is used as te cost metric. In general, any suitable function of m may be used as te cost. For instance, c(m) = K 1 + K 2 m, were K 1 and K 2 are some constants. Now we analyze te cost of messages during eac segment, wen using te competitive update protocol wit update limit L. In our analysis, we ignore te messages required to perform syncronization operations for obtaining access to a set of sared memory locations (i.e., acquire operation), as te cost of tese syncronization operations is typically independent of te update limit. As te nal goal of our analysis is to determine te optimal update limit, it is reasonable to exclude te syncronization costs. Let us assume te following notation: X denotes te average value of X. p update denotes size of an update message. p control denotes te size of a control message, suc as page request or acknowledgement. p page denotes te size of a message tat includes a page. Note tat p control and p page are constants, wile p update may be dierent for eac update. At rst, we consider message cost in a segment in wic te number of updates is U. A copy of te page is updated until it receives L update messages from oter nodes (between two consecutive local accesses). Upon receiving (L + 1)-t update message, local copy of te page is invalidated.

3 If te number of update messages (U) received during te segment is at most L (U L), ten te page is not invalidated. Terefore, te message cost of suc a segment is te cost of U update messages, and te U control messages acknowledging tese updates. Now consider te case wen te number of updates in te segment is at least L + 1. In tis case, te page will be invalidated on te (L + 1)-t update in te segment. Tis invalidation will result in a page fault wen a local access is attempted at te beginning of te next segment. Tus, a segment containing at least (L + 1) remote updates include two costs: (a) cost of L + 1 update messages and L + 1 control messages acknowledging tese updates, and (b) cost of serving a page fault. Cost of a page fault includes te following: (i) te request message, wic may be forwarded a few times before it reaces a node tat as te requested page, (ii) message to send te page to requesting node, and (iii) a control message to acknowledge receipt of te page. Te average cost, denoted C(L; U), of a segment wit U updates wen using competitive update protocol wit limit L is: i U c(p update ) + c(p control ), wen U L (L + 1) i c(p update ) + c(p control ) + (F + 1) c(p control ) + c(p page ), wen U > L. Note tat c(p control ) and c(p page ) are constants, wile c(p update ) depends on size of te update message, and F is te average number of ops a request message takes before reacing a node tat as te requested page. Terefore, te above expression uses average value of c(p update ). Now, let p(x) denote te probability density function (pdf) of te number of updates x in a segment for a given application. Based on te above cost analysis for a segment wit U updates and limit L, average cost per segment for te application, denoted C avg (L; p(x)), can be obtained as: C avg (L; p(x)) PL = c(p update ) + c(p control )i x=1 x p(x) + (L + 1) (c(p update ) + c(p control )) + (F + 1) c(p control ) + c(p page ) P 1 p(x) x=l+1 Using te average cost per segment for a particular limit L, optimal update limit, L optimal, can be obtained as te value of L tat minimizes te above expression for C avg (L; p(x)). More formally, L optimal = fl j min L C avg (L; p(x)) = C avg (l; p(x))g Now, in Section 4, we compares analytical results wit experimental measurements to validate te proposed model. 4 Validation of te Proposed Model To verify te accuracy of our cost analysis model, we compare te average cost estimated by our model wit tat obtained by experimental measurements. For tis validation study, we assume tat cost of a message of size m is m, tat is, c(m) = m. We implemented te competitive update protocol by modifying Quarks DSM (Beta release.8) [4, 9]. For tis study, we used several real applications (named Floyd-Warsall, Isort, Jacobi, SOR) as well as syntetic applications (named qtest5 and qtest1). qtest5 and qtest1 are simple sared memory applications based on a program available wit te Quarks release. All nodes access te sared data concurrently wit read ratio 5% and 1%, respectively. Floyd- Warsall is all-pair-sortest-pat algoritm. (We use 128 vertices as input.) Isort is Integer Sorting algoritm. (We use 3, keys of 1 range.) Jacobi is a linear system solver by using iteration metod. (We solve a linear system of size 128.) SOR is Successive Over-Relaxation algoritm wic executes simple iterative relaxation algoritm. (We use grid.) Floyd-Warsall, Isort, and Jacobi applications used in te experiments were written at Texas A&M University. SOR is available wit te Quarks release. Te experiments were used to determine: (a) te experimental cost for eac update limit L, and (b) te probability density function p(x) of te number of updates in a segment. All applications are executed on a 8-node workstation cluster. Using te p(x) measured by experiments, we ten computed te average cost per segment analytically. (We use F = 4 in tis analysis.) Te objective ere is to compare tis analytical result wit te cost measured experimentally. In Figures 2 troug 7, te curve labeled pdf plots p(x) 1, were p(x) is obtained via experimental measurements. For te pdf, te orizontal axis corresponds to number of updates x, and vertical axis corresponds to p(x) 1. p(x) is obtained by keeping track of te number of updates per segment for all pages at all nodes. Te cost.analysis curves in Figures 2 troug 7 plot te analytical values of average cost per segment, as obtained by using our model. Te cost.exp curve plots te average cost per segment as measured during te experiments. Te average is taken over all segments for all pages at all nodes. In our analysis, we did not consider te messages required for syncronization, initialization (wen te

4 pdf (%) or Cost (X 1^5 bytes) pdf (%) or Cost (X 1^2 bytes) Figure 2: qtest (read ratio = 5%) Figure 5: Isort (size = 3) pdf (%) or Cost (X 1^5 bytes) pdf (%) or Cost (X 1^4 bytes) Figure 3: qtest (read ratio = 1%) Figure 6: Jacobi (size = 128) pdf (%) or Cost (X 1^5 bytes) pdf (%) or Cost (X 1^5 bytes) Figure 4: Floyd-Warsall (size = 128) Figure 7: SOR (size = 512)

5 program starts executing), and false updates (sending update message to a node wose local copy as been already invalidated). Terefore, one would expect a mismatc between cost.exp and corresponding cost.analysis. However, te dierence between tese two curves would be relatively independent of te value of L. Te cost.adj (adjusted) curves in Figures 2 troug 7 are obtained by subtracting te unaccounted costs from cost.exp. Figures 2 troug 7 sow tat cost estimated by analytical model,cost.analysis, is typically close to te adjusted cost obtained by experimentation results, cost.adj. In fact, cost.analysis and cost.exp are also typically close. However, tere are signicant differences between cost.analysis and cost.exp, especially in te Figures 6 and 7. As noted before, tese dierences are due to unaccounted messages, for instance, for syncronization, initialization, and false updates. In case of Jacobi and SOR, many messages are sent initially to initialize te sared data. Tese messages form a large fraction of all messages sent. Terefore, te error in our model is muc more pronounced for tese applications, as compared to oter applications. To determine weter te probability density function varies muc wit te update limit, we measured p(x) by executing tree applications (qtest wit 5% read ratio, Jacobi, and SOR) for dierent update limits. Figure 8 troug 1 sow tat p(x) of qtest is relatively stable, p(x) of SOR is very stable, and te p(x) of Jacobi is relatively unstable (owever, in general, p(x) of Jacobi is decreasing as update limit increases). All 8 nodes sare some sared memory space in qtest and Jacobi applications. Te order of sared memory access may be dierent for eac loop in qtest application, and te order of nodes reacing te barrier may be dierent for eac loop of Jacobi application, wic causes p(x) to be unstable. However, only 2 nodes sare te sared memory space in SOR application wic causes p(x) to be stable. One may expect tat te probability density functions will be dierent wen using dierent input data for some applications. To verify, we measured te probability density functions of te number of updates in a segments wit dierent input data for Floyd- Warsall application. We coose Floyd-Warsall because memory access pattern is dierent on dierent inputs. (Te degree of saring increases if te input matrix is more dense.) Figure 11 sows te probability density functions of dierent input data for Floyd- Warsall application. Tere is only a little dierence for eac input data. pdf (%) pdf (%) pdf (%) pdf:l= pdf:l=2 pdf:l=4 pdf:l=6 pdf:l= Figure 8: pdf (qtest: read ratio = 5 %) pdf:l= pdf:l=2 pdf:l=4 pdf:l=6 pdf:l= Figure 9: pdf (Jacobi: size = 128) pdf:l= pdf:l=2 pdf:l=4 pdf:l=6 pdf:l= Figure 1: pdf (SOR: size = 512)

6 pdf (%) pdf:dense pdf:moderate pdf:sparse Figure 11: pdf (Floyd-Warsall: size = 128) Te above measurements sow tat for some applications, te probability density function p(x) is relatively independent of te update limit and input data. Of course, it sould be empasized tat, tis is not true for all applications. 5 Conclusion and Future Work Tis paper presents a new cost analysis model for competitive update protocol for software distributed sared memory (DSM). Tis model can be used to compute optimal update limit for te competitive update protocol. Te optimal limit is cosen suc tat te cost metric is minimized for te given application (as caracterized by its probability density function p(x) of number of updates x in a segment). We validated te proposed model by comparing analytical results from te model to experimental results obtained from an experimental DSM implementation. Te analytical results often closely matc te experimental results. We conclude tat te proposed model can, terefore, be used to estimate te optimal update limit for an application. Future work includes application of tis model to estimate costs of dierent scemes for recoverable DSM systems. Acknowledgments We tank te referees for teir detailed comments. We also tank Jon Carter and D. Kandekar at te University of Uta for making Quarks source code available in public domain, Akiles Kumar for te Floyd-Warsall source code, and Puneeta Rajan for Jacobi source code. References [1] C. Anderson and A. Karlin, \Two adaptive ybrid cace coerency protocols," in Proc. of te International Symposium on Hig-Performance Computer Arcitecture, pp. 33{313, Feb [2] M. Brorsson and P. Stenstrom, \Modeling accesses to migratory and producer-consumer caracterized data in a sared memory multiprocessor," in Proc. of IEEE Symposium on Parallel and Distributed Processing, pp. 612{619, Oct [3] M. Brorsson and P. Stenstrom, \Modeling accesses to stationary data in a sared memory multiprocessor," in Proc. of International Conference on Parallel and Distributed Computing Systems, pp. 82{87, Oct [4] J. Carter, D. Kandekar, and L. Kamb, \Distributed sared memory: Were we are and were we sould be eaded," in Proc. of te Fift Worksop on Hot Topics in Operating Systems, pp. 119{122, May [5] S. J. Eggers, \Simplicity versus accuracy in a model of cace coerency overead," IEEE Transactions on Computers, vol. 4, pp. 893{96, Aug [6] H. Gran, P. Stenstrom, and M. Dubois, \Implementation and evaluation of update-based cace protocols under relaxed memory consistency models," Future Generation Computer Systems, vol. 11, pp. 247{271, June [7] A. Karlin, M. Manasse, L. Rudolp, and D. Sleator, \Competitive snoopy cacing," in Proc. of te 27't Annual Symposium on Foundations of Computer Science, pp. 244{254, Oct [8] R. Kessler and M. Livny, \An analysis of distributed sared memory algoritms," in Proc. of te 9t International Conference on Distributed Computing Systems, pp. 498{55, June [9] D. Kandekar, \Quarks: Portable dsm on unix," Dept. of Computer Science, University of Uta (manuscript). [1] J.-H. Kim and N. H. Vaidya, \A cost-comparison approac for adaptive distributed sared memory," in Proc. of 1996 International Conference on Supercomputing, pp. 44{51, May [11] J.-H. Kim, Performance and Recoverability of Distributed Sared Memory Systems Using Competitive Update. P.D. dissertation, Texas A&M University, Aug [12] B. Nitzberg and V. Lo, \Distributed sared memory: A survey of issues and algoritms," IEEE Computer, vol. 24, pp. 52{6, Aug [13] A. Raynaud, Z. Zang, and J. Torrellas, \Distanceadaptive update protocols for scalable sared-memory multiprocessors," in Proc. of te International Symposium on Hig-Performance Computer Arcitecture, pp. 323{334, Feb [14] S. Srbljic, Z. Vranesic, and L. Budin, \Performance prediction for dierent consistency scemes in distributed sared memory systems," in Proc. of te International Symposium on Hig Performance Distributed Computing, pp. 295{32, Apr [15] M. Stumm and S. Zou, \Algoritms implementing distributed sared memory," IEEE Computer, vol. 23, pp. 54{64, May 199. [16] O. Teel and B. Fleisc, \Design and analysis of igly available and scalable coerence protocols for distributed sared memory systems using stocastic modeling," in Proc. of te International Conference on Parallel Processing, vol. I, pp. 126{13, Aug