Data Storage Institute. SANSIM: A PLATFORM FOR SIMULATION AND DESIGN OF A STORAGE AREA NETWORK Zhu Yaolong

Transcription

1 Data Storage Institute SANSIM: A PLATFORM FOR SIMULATION AND DESIGN OF A STORAGE AREA NETWORK Zhu Yaolong e_mail:zhu_yaolong@dsi.a-star.edu.sg

2 Outline Motivation Key Focuses Simulation Methodology SANSim Structure Workload Module Host Module Network Module Storage Module Calibration and Validation FC Network Analysis Summary Future Work

3 Motivation of SANSim New idea is everything. New algorithm is the key. The motivation of this research is to develop modeling and simulation capability and to use this capability to explore and research on new ideas and algorithms. Modeling and simulation are fast and effective way to conduct in-depth algorithm researches. There is lack of the such kind of methodology and tool in network storage industry. Storage system Block level FC/DWDM FC protocol FC/DWDM Storage system FC-SAN FCIP FCIP FC-SAN Server NAS iscsi WAN iscsi NAS Server Clients LAN Router Router LAN Clients File level IP protocol IP storage system IP storage system

4 SANSim Main Focuses Tools development Build up the capability of modeling and simulation of network and storage Simulation methodology Modular structure Algorithms research Modeling of storage architecture: DAS/NAS/SAN/OSD File system I/O workload analysis FC networks and their schedule algorithms Cache management Disk array design & algorithms (normal, degradation and rebuild mode)

5 Simulation Methodology Event-Driven simulation Trace-Driven System-Level Simulation Synthetic I/O Workload Virtual Device High-Resolution Timestamp Mechanism Globe Scheduling and queuing monitoring Closed System/Open System Models Warn-up processing Response time Warm up response time Simulation Duration simulation time Simulation time

6 SANSim Module Structure Clients I/O I/O Workload Storage Storage Clients Clients RAID Ctrl I/O Controller Requests Server Device Driver System Bus HBA Completions Host Host HBA FC Network Cache IP Network Network

7 SANSim Workload Generation Define the nature Parameters of I/O workload Synthetic workload Constant Interarrival time Generation of I/O workload Generation of requests Generate Workload & traces Typical pattern workload Sample application traces Examples Normally distributed Interarrival time Poisson Process Main functions for the workload generation Start Procedures of generating repeatable requests Exceed total request number? N Generate arrival time Select a document Generate a request action Y Form a complete request End

8 SANSim Host Modules Device Driver Controllers and HBA Buses Server I/O I/O Workload Clients Clients Clients Host Server Host Requests Completions Device Driver RAID Ctrl I/O Controller Host Host HBA Storage Cache Requests Device Driver Completions System Bus HBA FC Network Network IP Network System Bus HBA

9 SANSim Disk Module Data Cache/Buffer Disk Drive Module Request Processing & Control Requests Clients Mechanical Clients Positioning & Media Transfer Server Device Driver System Bus HBA I/O I/O Workload Clients Completions Host Host Storage RAID Ctrl I/O Controller HBA Cache Storage Storage Network FC Network IP Network Network Module Network Interface Data Layout & LBNto_PBN Mapping RAID Ctrl General Queue/ Scheduler Module I/O Controller HBA Cache

10 SANSim FC Networking Module Server Storage System FC Switch Server FC Switch Storage System FC Switch Server Storage System Server system FC Host Bus Adapter FC Connection FC ports FC Switch FC ports FC Connection FCP Target Controller Storage system FC Controller module FC Switch module Bus_interface FC Switch core Routing FCP Engine Internal Crossbar FC Port FC Port FC Port & Communication Frame buffer Management FC Connection (a)an example of FC SAN (b) Abstracted view of participating component (c) Simulation modules of the essential component s

11 SANSim Network Modules Bus interface Switch Fabric Cross-bar Routing FC Ports FC Controller Initiator FC Controller Target FCP Engine FC Port FC Network FC Port FCP Engine FC Port & Communication Module FC Port Frame buffer management & scheduling Connections - One to one - FCAL - Flow control - Link mgmt Bus interface

12 FCP Operation The rest of Server System The rest of Storage System Bus_interface FCP Engine FC port FCP_CMND FCP_DATA. FCP_XFER_RDY FCP_DATA FCP_RSP Bus_interface FCP Engine FC port initiator mode FC Port Communication module target mode

13 FC_port Module Disk Disk Inbound fibre Lport LPSM Outbound fibre FC-AL Disk Inbound FIFO FC-2 signal /framing control Outbound FIFO Disk Disk FCP Frame/Order Set level Host Fairness Access Algorithm Propagation delay due to physical distance FCP incoming Single Frame Queue FCP Data buffer FCP outgoing Single Frame Queue

14 Flow Control Module -- FC Frames Buffer Slots credit count 4 ++ R_RDY Credit based flow control FC Port A FC Port B Multi-stage Buffer-to- Buffer SRC BB Credits FC Frames SW FC Frames BB Credits SW FC Frames DST BB Credits Other traffic Other traffic

15 FC_Switch Modeling FC-SW FL-Port F-Port E-Port Input Channels Internal Crossbar output Channels Routing NL-Port N-Port FC Switch Input Buffers (a) FC_Switch module (b) FC_SW sub-module Feature of SANSim FC Switch Module: Load balancing on multi-link inter FC Switch connection FSPF routing/path selection Domain_ID for each Fibre Channel switch Area_ID and Port_ID for each N/NL port

16 IDC Model Validation Initiator Initiator FC-AL FC Analyzer Target Hardware CPU: AMD AthonMP FC HBA: Qlogic 2300 RAM: 2x256MB DDR SDRAM Mainboard: 64 bit PCI Tyan Tiger MP2466N Software Target Hardware OS: Windows XP Professional SP1 Driver: Qlogic Driver Version Tool: Intel IOMeter Version CPU: Intel PIII 1GHz FC HBA: Qlogic 2300 RAM: 4 x 1GB Kingston ECC Reg. PC133 Mainboard: 64bit PCI, Supermicro 370 Software OS: RedHat 8.0 Kernel: Driver: In-house 2300 target driver Ver 1.0, In-house Linux RAM Disk Ver 2.0 I/Os Per Second (k) Maximum IOPS vs. Queue Depth (Read) Tested. Simulated. 2KB 8KB 16KB 32KB Queue Depth Compare experimental data with simulation result Error range < 10%. Read operation <3% System configuration for testing experiment

17 Model Validation cont I/Os Per Second (k) Maximum IOPS vs. Request Size (Read) Tested Depth=8. Depth=2 Depth=1 Simulated. I/Os Per Second (k) Maximum IOPS vs. Request Size (Write) Depth=8 Depth=2 Depth=1. Tested. Simulated 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k Request Size 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k Request Size 200 Throughput vs. Request Size (Read) 200 Throughput vs. Request Size (Write) Throughput (MB/Sec Depth=8 Depth=1 Depth=2. Tested. Simulated Throughput (MB/Sec Depth=8 Depth=1 Depth=2. Tested. Simulated 0 0 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k Request Size 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k Request Size

18 FC Network Simulation and Analysis Servers 1 FC Switches ISL1 ISL8 ISL ISL7 ISL3 ISL6 ISL4 2 ISL Storages

19 Max Throughputs Impact Throughput (MB/Sec) a S1-4_32KB. S5-8_32KB. S1-4_2KB.. S5-8_2KB Throughput (MB/Sec) a S1-4_2KB S1-4_32KB S5-8_2KB S5-8_32KB Throughput (MB/Sec) a I/O load (MB/Sec) a a) No link failed S1-4_2KB S1-4_32KB S5-8_2KB S5-8_32KB Throughput (MB/Sec) a I/O load (MB/Sec) a b) ISL1 failed S1-4_2KB S5-8_2KB S1-4_32KB S5-8_32KB I/O load (MB/Sec) a c) ISL8 failed I/O load (MB/Sec) a d) Both ISL1 and ISL8 failed

20 Max Throughputs Impact Throughput (MB/Sec) a S1-4_32KB. S5-8_32KB. S1-4_2KB.. S5-8_2KB I/O load (MB/Sec) a 45MB/s for 2KB IO size, 80MB/s foe 32KB IO size. Performance is limited by network.

21 Max Throughputs Impact Throughput (MB/Sec) a S1-4_2KB S1-4_32KB S5-8_2KB S5-8_32KB I/O load (MB/Sec) a The asymptotic throughput of all servers drops. head-of-line blocking.

22 Max Throughputs Impact Throughput (MB/Sec) a S1-4_2KB S1-4_32KB S5-8_2KB S5-8_32KB I/O load (MB/Sec) a The storage link failure decreases the FC network throughput.

23 Max Throughputs Impact Throughput (MB/Sec) a S1-4_2KB S1-4_32KB S5-8_2KB S5-8_32KB I/O load (MB/Sec) a The link failure decreases the FC network throughput. The max throughputs are sensitive to network failure location.

24 Data Frame Latency Study Avg Latency (ms) Avg Latency (ms) Every each server (1-8) access to every each device(1-8) 0.0 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1m Request Size a) Case I: No link failure A: Each Server(1-8) to device 1-4 B: Each Server(1-8) to device 5-8 A Request Size REF Line 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1m B c) Case III: ISL8 failure Avg Latency (ms) Avg Latency (ms) A: Server 1-4 to each device B: Server 5-8 to each device A A: Server 1-4 to device 1-4 B: Server 5-8 to device 1-4 C: Server 1-4 to device 5-8 D: Server 5-8 to device 5-8 A Request Size C B Request Size REF Line 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1m D REF Line 0.0 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1m d) Case IV: Both ISL 1 and ISL 8 failure B b) Case II: ISL 1 failure The frame latency increases rapidly after knee point The frame latency is sensitive to network failure location. -7-

25 Summary SAN modeling - We have developed a simulation tool SANSim which supports FC frame level simulation. Model validation - We have conducted the experiments and compared experimental and simulated results. The results show that SANSim model is accurate with error range less than 3% for read operation. FC Network Simulation and Analysis -The link failure decreases the FC network throughput. The maximum throughputs are sensitive to network failure location. - The simulation results show that the performance loss can reach up to 50% when the servers runs with high workload. - The FC SAN network designers have to consider I/O workload requirements as well as the impact of network failure for critical application.

26 Future Work File system/osd simulation Application simulation IP Storage networking module Emulator Remote storage

27 Thank You For more technical information, please contact: Dr Zhu Yaolong ( Zhu_Yaolong@dsi.a-star.edu.sg)