iscsi Performance Evaluation and Improving of Sequential Storage Access using iscsi Protocol in Long-delayed High throughput Network

Transcription

1 SAN., ( ).,., OS ms.,.,., ( 1 ).,, Performance Evaluation and Improving of Sequential Storage Access using Protocol in Long-delayed High throughput Network Saneyasu YAMAGUCHI, Masato OGUCHI,andMasaruKITSUREGAWA Institute of Industrial Science, University of Tokyo KOMABA MEGURO-KU, TOKYO , JAPAN Research and Development Initiative, Chuo University 42-8 Ichigaya Honmura-cho, Shinjuku-ku Tokyo, , Japan {sane,kitsure}@tkl.iis.u-tokyo.ac.jp, oguchi@computer.org Abstract The protocol is becoming increasingly important as a convenient and cost-effective means of constructing Storage Area Networks (SANs). This paper details a comprehensive overview of performance issues pertaining to the protocol with specific focus on latency. In particular, we investigate the performance of for sequential accesses in high-latency high-throughput networks. Our performance evaluation demonstrates that the performance of degrades with increase in network latency. Moreover, we observe behaviors pertinent to sequential accesses and find evidence concerning the degradation in performance of owing to accesses associated with small block sizes. Further investigation of such behaviors revealed that accesses concerning large block sizes can be instrumental in improving the system performance significantly. Key words Storage System, Network Storage, 1., SAN(Storage Area Network. ),. SAN, DAS(Direct Attached Storage. ),., SAN (, FC-SAN ),, Ethernet TCP/IP

2 SAN (, IP-SAN ) SAN [1]. IP-SAN, SCSI over IP IETF [2], [3]., IP-SAN [4]. IP Storage ( 3. ),. ( 1 ),.,.,,... 2., IP-SAN. 3.,,., 4., SCSI. 5., SCSI, WWW,., SAN. SAN,. FC-SAN ( FC ),,, FC, FC. Ethernet TCP/IP SAN IP-SAN. IP-SAN,,,,,., Ethernet (FC 1Gbit/s ), ( - - ) IP. IP-SAN,SCSI, FC FCIP ifcp IETF [2], SNIA [6]. IP-SAN, FC-SAN FCIP ifcp., SCSI., TCP/IP, SCSI. SCSI (SCSI, SCSI, SCSI ) PDU(Protocol Data Unit), PDU TCP/IP. 2. 2, TCP/IP.,,,,..,., I/O [4], Ng SCSI over IP 8KB,., SCSIoverIP,. [5], Sarkar. CPU, TCP/IP.,. 3.,

3 PC Linux GBit Ether Initiator PC (FreeBSD Dummynet) GBit Ether GBit Ether Dummy delay Delay GBit Ether PC Linux dummy Target 2 2: CPU Pentium4 1.5GHz Main Memory 128MB OS Initiator, Target : Linux Dummnet : FreeBSD 4.5 RELEASE Gigabit Ethernet Card Network Interface Intel PRO/1 XT Server Adapter Initiator, Target : Gbit NIC 1 Dummynet : Gbit NIC 2 Sequential Read Throughtput[MB/sec] UNH IOL Draft 18 Initiator, Target reference implementation ver Byte MaxRecvDataSegmentLength MaxBurstLength Byte FirstBurstLength Byte Single Thread., Target..,, , Initiator( ) Target( ) Gigabit Ethernet TCP/IP. Ethernet, FreeBSD Dummynet [7]. Initiator Target, InterOperability Lab [8] reference implementation (, UNH ). 1. Initiator, Target, Dummynet PC, Initiator Target Linux, FreeBSD. PC 2., Initiator Target, Initiator OS(Linux) raw. Target.,.,. Throughtpu[MB/sec] cross ms 1ms 2ms 4ms 8ms 16ms 1 Way Dealy (UNH) 2 :, 5KB. TCP Window Size 1MB (1 Way Delay), Initiator Target. ms, 1ms, 2ms, 4ms, 8ms, 16ms (Dummynet) Initiator Target. ms Dummynet. ms 14µs. (UNH),., (UNH) (, TCP Window Size ), ( ) (, ). 5KB 1, TCP Window Size 1MB. OS raw read (, 4. 4 ). 4[MB/sec], Dummynet., Target Initiator,., (UNH) 1 5KB 5 124B

4 ,.. 4.,,.,, ( ),., 3. Linux OS ( 2 (UNH) ),., 4ms (Dummynet ), 8ms. TCP Window Size 2, (TCPWindowSize)/(2 ). TCP Window Size( Window Size), ( )Ack. TCP Window Size TCP, 16bit., Window Size 64KB, TCP Window Scaling (RFC 1323) [9] 1GB. Window (TCPWindowSize)/(2 ). Window Window, Window SCSI TCP/IP, SCSI over TCP/IP (over Ethernet) 2., TCP/IP.,,. 2 PDU SCSI CDB, SCSI over PDU SCSI CDB Initiator Target 3 Read Idle Data-in Data BlockSize TransferTime Idle TCP Ack Idle 1 Cycle = 4*OneWayDelay + TransferTime Response Idle Time, TCP/IP, Ethernet, TCP/IP,., TCP Window Size TCP. 4. 3,. read., raw 5KB read. read, 3., Initiator Target Read PDU, Target ( 3 Read ). Target Read Data-in PDU Initiator, Initiator ( 3 Data-in Data )., Initiator Target TCP Ack, Target ( 3 TCP Ack )., TCP Ack Initiator TCP ( Data-in TCP ) TCP/IP,., Target Response PDU Initiator, Initiator ( 3 Response ), 1 Read., (, Seq. Read ). 3, Read 1 4 ( Idle ),.,, 8ms

5 InitiatorHost 1 Cycle TargetHost Time[sec] Packet Ack 91 Packets 4 TCP Packet Initiator Target Read 48B IP Header 2 Byte Data-in 48B 2Byte Data 127.5KB TCP Header 32 Byte N 2 TimeStamp Option 1Byte 15 Byte TCP Ack B Response 48B TCP Packet Payload 1448 Byte Time 5 TCP Packet 8ms, TCP Window Size 1MB,. 5KB read. I T Initiator Target, T I Target Initiator. 4, T I :Data-in 91, I T :Read, I T : TCP Ack, T I : Response 1. T I : Data-in 91 TCP, (TCP/IP ) 1368 Byte, 48Byte 127.5KB. 5KB 127.5KB, TCP/IP. SCSI over TCP/IP over Ethernet, SCSI PDU TCP/IP. TCP, 1 TCP 1., 1 PDU 1 ( i Header, idatasegment Data Segment.).. TCP/IP. 5 TCP Ack, Byte TCP/IP TCP. Ack TCP., ( PDU) TCP/IP, TCP/IP TCP TCP/IP Ethernet. Data-in. 1 Target 127.5KB. 2 (48Byte) PDU(127.5KB+48B). 3 TCP/IP PDU MSS(Maximum Segment Size : ), TCP/IP TCP/IP. Ethernet MTU(Maximum Transmission Unit. 1 ) 15, TCP/IP 52, MSS TCP/IP 2 IP, 2 TCP, 12 TCP (Nop, Nop, Time Stamp Option). 4 Ethernet TCP/IP ( 15 ) 14 Ethernet 4,. Seq. Read, Initiator Read PDU. 1 1 = =4 + (1).,., = (3) 4 +., TCPWindowSize 2.,. TCP/IP < TCPWindowSize = 2 (4) = (5) 4 + (2) 3. 2 (UNH), 1ms 9%, 2ms 8%, 4ms 4%, 8ms 1%, 16ms 4%., 4. 4

6 Application Read 5KB File System Block Device Character Device SCSI Generic Driver Driver TCP/IP NIC Driver Network NIC Hardware Initiator side Target side 6 125KB.,,, TCP Window Size,. 4. 4, 3, , LinuxOS, raw read. OS, read,, SCSI, Initiator, TCP/IP (, Read PDU).,. ( 3. 1 ), 6, Target PDU raw 5KB read, KB 4 Read PDU KB 5 Read PDU 1PDU Target ( OS Target PDU Read ). Seq. Read, OS read 3, 4 SCSI SCSI ms 2ms 4ms 8ms 16ms Seq. Read ( ) 3.,., Read, 3. ( 125KB).,. 5.,,. 5. 1, 2. 1 TCP Window Size, TCP/IP. 2 Read Block,. (1)TCP Window Size, TCP/IP., (2) Block, 3 3 Seq. Read, (2) (,. 512 )., MaxBurstLength Data-in PDU

7 Read PDU Initiator. Initiator,, Target TCP/IP. UNH draft 18 [3]. Initiator, draft 18, login, SCSIRead., Login Request PDU Login Parameter(MaxRecvDataSegmentLength, MaxBurstLength, Frist- BurstLength ), SCSI Read Command PDU, SCSI Data-in PDU, SCSI Response PDU, Read (Initiator Task Tag, Command Sequence Number, Status Sequence Number). ErrorHandling., 3. 1., (1), (2), (3) Initiator 3, (1), (2) (UNH), (3) (KI). (1) Initiator Target. (2) Initiator Target raw. Initiator, Target UNH (Target ). OS SCSI Initiator. (3) Initiator Initiator, Target PDU. Initiator Initiator, Target UNH Target ( Target ). Initiator, PDU OS SCSI,. (2) (3),., (2) (3) 1..., 3. Initiator 125KB, 1MB, 2MB, 4MB 7., (UNH) 8. (UNH), 3. 2, (UNH) 5KB, PDU Read 125KB. 8 Ratio (UNH) (KI) ( (UNH) ) (KI). (UNH) (KI). Throughput[MB/sec] ms 1ms 2ms 4ms 8ms 16ms 1 Way Delay (UNH) (KI) 125K (KI) 1M (KI) 2M (KI) 4M 7 : Initiator UNH Initiator 5KB, 125KB PDU. Initiator 125KB, 1MB, 2MB, 4MB. TCP Window Size 1MB. Throughput[MB/sec] Ratio[] ms 1ms 2ms 4ms 8ms 16ms 1 Way Delay Ratio[U/K125K] Ratio[U/K1M] Ratio[U/K2M] Ratio[U/K4M] : [UNH/ Initiator] (UNH) (KI) 2KB 4KB 8KB 16KB 32KB 64KB 128KB 256KB 512KB 1MB 2MB 4MB Block Size[Byte] 9 16ms, TCP Window Size 1MB,.,, Seq. Read

8 ., 9., (UNH) read 128KB, Initiator. (UNH).941[MB/sec], (KI) 1.4[MB/sec],, ( 1 )., 16ms 1.4[MB/sec],., 2. ms, Window Size 1MB. 6.., OS., Read., Initiator,. ms,., SAN. 4ms, 16ms 1[MB/sec].,.,,,. [1],,, 22 [2] IETF : [3] IETF IPS, /draft-ietf-ips-iscsi-19.txt [4] Wee Teck Ng, Bruce Hilly Elizabeth Shriver, Eran Gab- Throughput[MB/sec] ms 1ms 2ms 4ms 8ms 16ms 1 Way Dealy A 1 Throughput[MB/see] A Ratio[] (UNH) (KI) Ratio[U/K] (UNH) (KI) (KI) 8KB 16KB 32KB 64KB 128KB 256KB 512KB 1MB 2MB 4MB Block Size[Byte] ( ms, Window Size 1MB) ber, Banu Ozden, Obtaining High Performance for Storage Outsourcing, Proc. FAST 22, USENIX Conference on File and Storage Technologies, January 28-29, 22, pp [5] Prasenjit Sarkar and Kaladhar Voruganti, IP Storage: The Challenge Ahead, Proc. of Tenth NASA Goddard Conference on Mass Storage Systems and Technologies, April 22 [6] SNIA : [7] L. Rizzo, dummynet, luigi/ip dummynet/ [8] The University of New Hampshire s InterOperability Lab linux.html [9] RFC 1323 TCP Extensions for High Performance 1. A 1 ( 64KB) (UNH) (KI) (Window Size 1MB)., SCSI ( 5. 2 ). 2. ms, Window Size 1MB Initiator A , ms 14µs., Initiator.