A Network Bandwidth Computation Technique for IP Storage with QoS Guarantees

Transcription

1 A Network Bandwdth Computaton Technque for IP Storage wth QoS Guarantees Young Jn Nam 1, Junkl Ryu 1, Chank Park 1, and Jong Suk Ahn 2 1 Department of Computer Scence and Engneerng Pohang Unversty of Scence and Technology/PIRL Kyungbuk, Republc of Korea {yjnam,lancer,cpark}@postech.ac.kr 2 Department of Computer Engneerng Dongguk Unversty, Seoul, Republc of Korea jahn@dgu.ac.kr Abstract. IP storage becomes more commonplace wth the prevalence of the SCSI (Internet SCSI) protocol that enables the SCSI protocol to run over the exstng IP network. Meanwhle, storage QoS that assures a requred storage servce for each storage clent has ganed n mportance wth ncreased opportuntes for multple storage clents to share the same IP storage. Consderng the exstence of other competng network traffc n IP network, we have to provde storage I/O traffc wth guaranteed network bandwdth. Most mportantly, we need to calculate the requred network bandwdth to assure a gven storage QoS requrement between a storage clent and IP storage. Ths paper proposes a network bandwdth computaton technque that not only accounts for the overhead caused by the underlyng network protocols, but also guarantees the mnmum data transfer delay over the IP network. Performance evaluatons wth varous I/O workload patterns on our IP storage testbed verfy the correctness of the proposed technque; that s, allocatng a part (0.6 20%) of the entre network bandwdth can assure the gven storage QoS requrements. 1 Introducton Storage Area Networks (SAN), such as Fber Channel and Ggabt Ethernet, have enabled a plethora of storage systems to be mantaned as a storage pool, resultng n reduced total cost of ownershp, effectve storage resource management, etc. Such SAN-based storage systems are advantageous n terms of scalablty and confgurablty, compared wth SCSI bus-based storage systems. Accordngly, a few data transmsson protocols have newly emerged to support the SAN envronment. Fber Channel protocol (FCP) s developed for FC-based SAN, and SCSI recently ratfed by Internet Engneerng Task Force s made for IP SAN. A man advantage of SCSI s that the SCSI can operate on standard network components, such as Ethernet [1]; that s, t explots exstng features and tools that have been developed for the IP network. Thus, ths paper focuses H. Jn et al. (Eds.): NPC 2004, LNCS 3222, pp , c IFIP Internatonal Federaton for Informaton Processng 2004

2 474 Y.J. Nam et al. on the storage envronment usng IP-based SAN (IP storage), where storage devces are attached to IP networks, and storage clents communcate wth the storage devces va the SCSI protocol [1]. An ntator-mode SCSI protocol runs on the storage clent, whereas a target-mode SCSI protocol operates on the IP storage. Note that the tradtonal SCSI protocol operates on top of the SCSI protocol layer that transmts a gven SCSI command to ts assocated IP storage over the IP network. Wth the advance n storage technologes n terms of storage space and I/O performance, the chances ncrease that multple storage clents share the same storage. A dfferent storage clent may requre a dfferent storage servce, called storage Qualty of Servce (QoS); that s, each storage clent requres recevng a guaranteed storage servce, ndependently of the status of the I/O servces n other storage clents. Unfortunately, the storage tself does not contan any feature of assurng storage QoS. As a result, recent research efforts [2,3] try to add the QoS feature to varous types of storage systems. However, notce that the prevous research emphaszes the QoS ssue only wthn the storage system, whereas t s assumed that the SAN tself has no QoS ssues. Note that, FCbased SAN s used only for storage I/O traffc. IP storage, however, transmts ts data over the IP network, where storage I/O traffc s lkely to coexst wth the other network traffc. Consderng ths stuaton leads us to preserve an amount of network bandwdth for the storage I/O traffc between a storage clent and ts assocated IP storage to avod any probable nterference wth the other network traffc. A nave approach s to allocate the full network bandwdth (or separate dedcated IP network) that s large enough to serve the storage I/O traffc wth QoS guarantee for a par of a storage clent and ts assocated IP storage. However, t can be easly nferred that ths approach ends up beng wth under-utlzaton of IP network resources, even though t can certanly guarantee a gven storage QoS requrement. By contrast, unless enough network bandwdth resdes between the storage clent and the IP storage, the storage QoS requrement s no longer guaranteed wth lower I/O throughput (I/O requests per seconds) and hgher response tme due to ncreased data transfer delays. Ths paper emphaszes the problem of computng the requred network bandwdth to meet a gven storage QoS requrement. It proposes a network bandwdth computaton technque that not only accounts for overhead caused by the underlyng network protocols, but also guarantees the mnmum data transfer delay over the IP network. In the case of FC-based SAN envronments, Ward et. al. n [4] proposed a scheme to automatcally desgn an FC-based SAN that not only serves a gven set of storage I/O traffc between storage clents and storage systems, but also mnmzes the system cost. 2 Problem Descrpton We begn by defnng a few notatons to be used throughout the paper. C and Q represent the storage clent and ts storage QoS requrement, respectvely. Generally, the storage QoS requrement s defned as Q = {f,ops,sz,s,rt }.

3 A Network Bandwdth Computaton Technque for IP Storage 475 The notaton of f represents the rato of read I/O requests. In addton, ops, sz,andrt represent the number of I/O requests per second (brefly IOPS), an average I/O request sze, and an average response tme requested from C, respectvely. I/O access pattern s random wth s = 0 and purely sequental wth s = 1. Note that our network bandwdth computaton does not depend on the value of s.wedenotewthb and b the network bandwdth allocated for the drecton from C and ts assocated IP storage and for ts opposte drecton, respectvely. Then, the problem that ths paper wll solve can be descrbed as follows: Compute b and b that satsfy the gven Q for the storage clent C and ts assocated IP storage. Note that we assume that the storage resources except for network resources (bandwdth) have been approprately reserved to satsfy the storage QoS requrement Q. 3 The Proposed Technque To begn, we wll explan the protocol layerng for SCSI protocol and the specfc behavor of the SCSI protocol for read and wrte I/O requests. The SCSI protocol data unt (PDU) conssts of a 48-byte SCSI header and SCSI data of a varable length. The maxmum SCSI data length reles on the types of underlyng Ethernet cards. Typcally, t ranges from 1,394 through 8,894 bytes. The TCP/IP headers and Ethernet header respectvely occupy 40 bytes and 18 bytes. Fgure 1 presents the protocol behavors for read and wrte I/O requests. In the case of the read I/O request, as shown n Fgure 1(a), the storage clent sends the READ SCSI command to the IP storage. Next, after readng the requested data from ts nternal dsk drves, the IP storage transmts the data to the storage clent n the DATA IN phase. Note that the data are to be fragmented nto smaller peces accordng to the maxmum SCSI data length. Fnally, the IP storage sends the response message to the storage clent. The SCSI CMD-Read PDU (106B) DATA_IN PDU..... DATA_IN PDU CMD-Resp PDU (106B) CMD-Wrte PDU (106B) (w/ DATA_OUT PDU) DATA_OUT PDU..... DATA_OUT PDU R2T PDU (106B) DATA_OUT PDU CMD-Resp PDU (106B) (a) read (b) wrte Fg. 1. SCSI protocol behavors for read and wrte I/O requests: (a) read and (b) wrte

4 476 Y.J. Nam et al. protocol behavor for the wrte I/O request s more complcated than that of the read I/O request because of free buffer management and performance optmzaton technques lke mmedate/unsolcted data transmsson. The storage clent sends the WRITE SCSI command to the IP storage. Unless the data sze s greater than the maxmum SCSI data length, t s transmtted along wth the wrte SCSI command. Ths results n collapsng the COMMAND phase and the DATA OUT phase [1]. It s called mmedate data transmsson [1]. If the data sze s greater than FrstBurstLength, the storage clent transfers the data of the frst FrstBurstLength bytes to the IP storage wthout recevng the Ready to Transfer (R2T) message from the IP storage that s used to secure free buffer space to store the wrte data. Ths process s called unsolcted data transmsson [1]. Note that the mmedate data transmsson s combned wth the unsolcted data transmsson. Afterwards, the storage clent transfers data only when t receves the R2T message from the IP storage. It s called solct data transmsson [1]. In what follows, we derve a set of equatons to compute the requred network bandwdth to meet a gven storage QoS requrement. We start by computng the amount of data transfer, ncludng the underlyng protocol overhead for gven read and wrte I/O requests of sze s.wedenotewthdr (sz ) the amount of data transfer from the storage clent to the IP storage for the read I/O request of sze sz,anddr (sz ) for the opposte drecton. From the protocol behavor for the read I/O request, as shown n Fgure 1(a), we can easly obtan Dr (sz ) and Dr (sz ) as follows: Dr (sz )=ov prot, (1) Dr (sz )=2ov prot + sz S f + sz mod Ŝ Ŝf, f (2) where Ŝf = S f ov prot. The notaton of ov prot represents the 106-byte protocol overhead caused by the Ethernet header, the TCP/IP header, and the SCSI header. The notaton of S f represents the underlyng Ethernet frame sze. Next, we calculate the amount of data transfer for the wrte I/O request. To begn, the storage clent sends the agreed-upon amount of data (unsolcted data transmsson) to IP storage wthout havng an R2T message. We assume that the behavor of each data transfer follows that of the solct data transmsson because the network traffc under our consderaton s heavy enough. However, f the I/O request sze s not greater than the maxmum SCSI data length, the assocated data s delvered to the IP storage by usng the mmedate data transmsson. As wth the read I/O request, we denote wth Dw (sz ) the amount of data from the storage clent to the IP storage for the wrte I/O request, and Dw (sz ) for the opposte drecton. Based on the SCSI protocol behavor for the wrte I/O request, as shown n Fgure 1(b), Dw (sz )anddw (sz )can be obtaned as follows: { Dw (sz ovprot + sz f sz )= Ŝf, (3) ov prot + sz S Ŝ f + sz mod Ŝf otherwse f

5 A Network Bandwdth Computaton Technque for IP Storage 477 { Dw ovprot f sz (sz )= FrstBurstLength. (4) 2ov prot otherwse If the data sze s not greater than the maxmum SCSI data length of Ŝ f, Dw c z(sz )sov prot + sz. Otherwse, t ncludes sz tmes of the maxmum Ŝ f Ethernet frame sze of S f and the remanng data wth the protocol overhead of ov prot.asfordw (sz ), f the data sze s not greater than FrstBurstLength, Dw (sz ) s equal to the sze of the SCSI response message. Otherwse, t becomes two tmes of ov prot, because the R2T message s also delvered to the storage clent as well as the SCSI response message. Next, we defne wth the average amount of data transfer ncludng the underlyng protocol overhead from the storage clent to the IP storage for the sz szed I/O request. In addton, we defne wth for the opposte drecton. From Equaton (1) (4), we can obtan and as follows: = {f Dr (sz )+(1 f )Dw (sz )}ops, (5) = {f Dr (sz )+(1 f )D w (sz )}ops. (6) and s expected to assure the The network bandwdth allocaton wth requested maxmum storage bandwdth derved by multplyng sz and ops. However, they may not guarantee the demanded response tme of rt. For example, notce that Equaton (5) and (6) result n a lower network bandwdth wth a smaller sz ops. Ths mples that the chances ncrease that each I/O request of sze sz experences a longer transmsson delay on IP network that s most lkely to ental a volaton of the demanded response tme. Thus, we ntroduce the mnmum network bandwdth to assure the demanded response tme of rt.wedenotewthm and m the mnmum network bandwdth for each drecton. The values of m and m are determned such that the transmsson delay of each I/O request are not greater than α rt, where 0 <α < 1. Usually, α s determned accordng to the margnal response tme to meet rt n the phase of desgnng the assocated IP storage [2,5,3]. For example, f the IP storage s desgned to assure 15msec for gven rt = 20msec, the values of α can range from 0 through We compute m and m from a smple relatonshp that the expected transmsson delay s nversely proportonal to the allocated network bandwdth wthout accountng for the effects of traffc congeston control, IP routng, the transmsson buffer sze, TCP retransmsson, etc. The m and m m m By denotng wth b are wrtten as follows: = max{ f Dr = max{ f Dr and b the network bandwdth for each drecton re- and b as follows: qured to guarantee a gven Q, we fnally have b b b (sz ), 1 f Dw (sz )}, (7) α rt (sz ), 1 f Dw (sz )}. (8) α rt = max{,m }, (9) = max{,m }. (10)

6 478 Y.J. Nam et al. 4 Performance Evaluatons We set up an expermental testbed for IP storage to evaluate the performance of the proposed technque.we use two Intel Pentum III based desktops for the storage clent and the IP storage. Both systems are attached to a Ggabt IP network va Ggabt Ethernet cards and a swtch. Assume that no other traffc exsts between the two systems. The maxmum sze of the Ethernet frame s 1500 bytes. The Lnux kernel works on top of the storage clent and IP storage. The storage clent ncludes the ntator-mode SCSI drver developed by the Unversty of New Hampshre [6], and the IP storage contans the targetmode SCSI drver for operatng the SCSI protocol. The network bandwdth s controlled by Token Bucket Flterng (TBF) [7]. We beleve that ths type of end-to-end traffc control works because no other traffc exsts between the storage clent and the IP storage. Table 1 shows the measured ops and rt for varous QoS requrements of Q 1 Q 4 when the amount of network bandwdth s computed by the proposed technque. In addton, they are compared wth the results wth the full network bandwdth for the same QoS requrements. We denote wth prot-rt and full for the proposed technque and the full network bandwdth, respectvely. Each Table 1. Results of ops and rt for the varous QoS requrements of Q 1 Q 8 by the proposed technque (prot-rt) and the full network bandwdth allocaton (full) QoS Technque Toward IP storage Toward clent ops rt Q 1 prot-rt 0.05MB/s 0.60MB/s 84(100%) 9.74ms full MB/s 100MB/s 82(100%) 9.70ms Q 2 prot-rt 1.84MB/s 0.17MB/s 265(99%) 2.59ms full MB/s 100MB/s 266(100%) 2.58ms Q 3 prot-rt 0.03MB/s 19.19MB/s 77(100%) 15.66ms full MB/s 100MB/s 76(100%) 15.34ms Q 4 prot-rt 15.67MB/s 0.02MB/s 211(100%) 20.28ms full MB/s 100MB/s 208(100%) 21.54ms of the storage QoS requrements s represented as follows: Q 1 = {f 1 =1,ops 1 = 82,sz 1 = 1KB,s 1 = 0,rt 1 = 10ms}, Q 2 = {f 2 = 0,ops 2 = 266,sz 2 = 1KB,s 2 =0,rt 2 =3ms}, Q 3 = {f 3 =1,ops 3 =76,sz 3 =64KB,s 3 =0,rt 3 = 18ms}, andq 4 = {f 4 =0,ops 4 = 208,sz 4 =64KB,s 4 =0,rt 4 =22ms}. The frst two requrements are for the small read and wrte I/O requests as n OLTP applcatons, and the others are for the large read and wrte I/O requests n scentfc applcatons [5]. Snce the network bandwdth allocaton s ndependent of storage access patterns, we assume that all the storage access patterns are purely random. In addton, the demanded IOPS of ops and ts assocated response tme of rt are confgured by njectng a set of I/O workload patterns and mea-

7 A Network Bandwdth Computaton Technque for IP Storage 479 surng each performance. Note that the ops and rt of the QoS requrements fall nto nether too lght traffc that makes the IP storage mostly dle nor too heavy traffc that overloads the system. The results reveal that the allocaton of only % of the full network bandwdth computed by the proposed technque can meet the gven storage QoS requrements. Observe that the proposed technque can compute an approprate amount of network bandwdth to provde the same qualty of storage servce as the case when the full network bandwdth s allocated. The percentage values n the ops column represent the percentage of the measured IOPS wth respect to the demanded IOPS. Table 2 shows the measured ops and rt for the QoS requrements of Q 1 Q 4 for the prot and nave technques. The prot technque computes the requred network bandwdth only by consderng the underlyng protocol overhead, as shown n Equaton (5) (6); that s, and.thenave technque smply calculates the network bandwdth by multplyng ops and sz. It assgns the same bandwdth for each drecton. As expected, the nave technque cannot guarantee even demanded ops, because t does not account for the underlyng protocol overhead n Ethernet, TCP/IP, and SCSI layers at all. Notce that Table 2. Results of ops and rt for the varous QoS requrements of Q 1 Q 8 by the technque usng Equaton (5) (6) (prot) and the technque usng ops sz (nave) QoS Technque Toward IP storage Toward clent ops rt Q 1 nave 0.06MB/s 0.06MB/s 64(77%) 28.82ms prot 0.01MB/s 0.12MB/s 80(96%) 12.99ms Q 2 nave 0.26MB/s 0.26MB/s 238(89%) 9.25ms prot 0.29MB/s 0.03MB/s 262(98%) 3.54ms Q 3 nave 4.78MB/s 4.78MB/s 69(91%) 22.31ms prot 0.01MB/s 5.19MB/s 74(96%) 19.33ms Q 4 nave 13.06MB/s 13.06MB/s 194(93%) 25.13ms prot 14.08MB/s 0.02MB/s 208(99%) 22.05ms the smaller szed I/O request causes hgher protocol overhead, as expected from Equaton (1) (4). In addton, the results show that read I/O requests creates more protocol overhead than wrte I/O requests. It can be expected manly from Equaton (2) (3). By contrast, the prot technque guarantees more than 96% of the requred ops. However, t does not satsfy the demanded response tme of rt because of the transmsson delay on IP network. Recall that, as shown n Table 1, the proposed prot-rt technque can meet both the ops and the rt by effectvely allottng the network bandwdth for each drecton between the storage clent and the IP storage.

8 480 Y.J. Nam et al. 5 Concluson and Future Work Ths paper addressed the problem of effectvely allocatng network bandwdth to assure a gven QoS requrement for IP storage. It defned a specfcaton of the storage QoS requrement, and t proposed a technque to compute the demanded network bandwdth to meet the storage QoS requrement that not only accounts for the overhead caused by the underlyng network protocols, but also guarantees the mnmum data transfer delay over the IP network. Performance evaluatons wth varous I/O workload patterns on our IP storage testbed verfed the correctness of the proposed technque; that s, allocatng a part (0.6 20%) of the entre network bandwdth can assure the gven storage QoS requrements. Currently, we have been revsng Equaton (9) (10) to addtonally account for real-world I/O workload patterns featured by self-smlarty and the condton of traffc congeston. Acknowledgments The authors would lke to thank the Mnstry of Educaton of Korea for ts fnancal support toward the Electrcal and Computer Engneerng Dvson at POSTECH through ts BK21 program. Ths research was also supported n part by the Korea Scence and Engneerng Foundaton (KOSEF) under grant number R and by HY-SDR IT Research Center. References 1. Meth, K., Satran, J.: Desgn of the scs protocol. In: Proceedngs of the Mass Storage Systems and Technologes/20th IEEE/11th NASA Goddard Conference. (2003) 2. Nam, Y.J.: Dynamc Storage QoS Control for Storage Cluster and RAID Performance Enhancement Technques. Ph.D Dssertaton, POSTECH (2004) 3. Anderson, E., Hobbs, M., Keeton, K., Spence, S., Uysal, M., Vetch, A.: Hppodrome: Runnng rngs around storage admnstraton. In: Proceedngs of Conference on Fle and Storage Technologes. (2002) 4. Ward, J., O Sullvan, M., Shahouman, T., Wlkes, J.: Appa: Automatc storage area network desgn. In: Proceedngs of Conference on Fle and Storage Technologes. (2002) 5. Alvarez, G., Borowsky, E., Go, S., Romer, T., Becker-Szendy, R., Goldng, R., Merchant, A., Spasojevc, M., Vetch, A., Wkes, J.: Mnerva: An automated resource provsonng tool for large-scale storage systems. ACM Transactons on Computer Systems 19 (2001) UNH: scs reference mplementaton. (2004) 7. Hurbert, B.: Lnux advanced routng & traffc control. (2003)