IP Storage Network Design

Transcription

1 IP Storage Network Design Session 2 Copyright Printed in USA.

2 Agenda Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 3 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 4 Copyright Printed in USA.

3 The Typical Storage Environment Direct Attached Storage (DAS) Storage is captive behind the server Server CPU must handle user I/O requests, but also: User database inquiries User file/print serving Data integrity checking Communication with other devices Data access is file system and platform dependant Costly to scale; complex to manage Clients IP Network Servers Win2k Linux Unix Win2k Linux Unix FC FC Direct Attached Storage SCSI 5 Storage System Components Application Server Connection to Clients (IP/Ethernet) Just-a-Bunch-of- Disks (JBOD) Disk Subsystem Connection to Servers (SCSI, FC, ) Internal Storage Connection to Additional Storage (SCSI, FC, ) Commonly have internal storage for host OS Direct-attached or networkattached to additional storage Front-end IP connection for client communication JBOD are simple groups of disks with no data protection (no RAID) JBODs are commonly Parallel SCSI or Fibre Channel Loop attached Disk Subsystems are complex arrays of disks with many services (RAID) 6 Copyright Printed in USA.

4 The SCSI I/O Channel SCSI is the protocol used to communicate between Servers and Storage devices SCSI I/O Channel provides a half-duplex pipe for SCSI CDBs and Data Parallel implementation Bus width: 8, 16 bits Bus speed: 5 80 Mhz Throughput: MBps Devices/bus: 2 16 devices Cable length: 1.5m 25m A network approach can scale the I/O channel in many areas (length, devices, speed) SCSI CDB: SCSI Command Descriptor Block Used to Relay SCSI Commands, Parameters, and Status between SCSI Initiators and SCSI Targets; Typically 6, 10, or 12 Byte Block TCP/IP Stack NIC Driver NIC Adapter Applications File System Block Device SCSI Generic Adapter Driver SCSI Adapter SCSI Initiator SCSI Half-Duplex SCSI I/O Channel SCSI Target SCSI SCSI 7 Networking the I/O Channel Same SCSI protocol (SCSI-3) carried over a network transport via serial implementation Transport must not jeopardize SCSI payload (security, integrity, latency) Two primary transports to choose from today: namely IP and Fibre Channel A networked I/O channel allows for multiple improvements: Distance limitations greatly increased Dedicated bandwidth (not shared) High # of addressable devices Bandwidth increase (including link bundling) Initiator Networked I/O Channel Host System SCSI Channel Controller Network Target and LUNs 8 Copyright Printed in USA.

5 Considerations for Networked I/O There are several necessary considerations for networking the I/O channel: Security: Ensuring the appropriate initiators access appropriate targets Performance: Ensuring the networked channel has adequate bandwidth and flow control Latency: Ensuring the channel does not experience intolerable latency compromising application integrity Availability: Ensuring the networked channel can recover from topological faults in a timely manner Resource Management: Ensuring network resources can be monitored and accounted for Fault Management: Ensuring the proper tools exist to troubleshoot the chosen network Interoperability: Ensuring the network can be expanded without jeopardizing network stability Initiators Network Targets and LUNs 9 Fibre Channel Networking Very common method for networking SCSI Fibre Channel provides high-speed transport for SCSI payload Fibre Channel overcomes many shortcomings of DAS including: Addressing for up to 16 million nodes (24 bits) Loop (shared) and Fabric (switched) transport Speeds of 100 or 200 MBps (1 or 2 Gbps) Distance of up to 10km (without extenders) Support for multiple protocols Combines best attributes of a channel and a network Initiator Host System SCSI Fibre Channel HBA Fibre Channel Fabric Target 10 Copyright Printed in USA.

6 IP An Alternate I/O Transport Host System Viable transport for I/O traffic Not necessarily for long-haul I/O only Similar characteristics to Fibre Channel: Addressing for close to 4 billion nodes (IPv4) Primarily a switched transport (with routing) Ethernet speeds of 10/100/1000 Mbps (10 Gbps arriving) or various WAN speeds Support for multiple high level protocols Cost and manageability advantages with IP IP knowledge base widespread in industry Initiator SCSI IP Channel Adapter IP Network Target 11 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 12 Copyright Printed in USA.

7 IP Storage Networking IP storage networking provides solution to carry storage traffic within IP Uses TCP: A reliable transport for delivery Can be used for local data center and long haul applications Two primary protocols: IP-SCSI Used to transport SCSI CDBs and Data within TCP/IP connections IP TCP SCSI Data FCIP Fibre-Channel-over-IP Used to transport Fibre Channel frames within TCP/IP connections Any FC frame Not just SCSI IP TCP FCIP FC SCSI Data 13 What Is? A SCSI transport protocol that operates over TCP Encapsulates SCSI CDBs and data into TCP/IP byte-streams (defined by SAM-2 SCSI Architecture Model 2) Allows IP hosts to access IP based SCSI targets (either natively or via to FC router) Standards status IETF Internet Draft passed last call (draft 20 released January 2003) Current draft: draft-ietf-ips-iscsi-20 Internet Standard RFC expected Q2 14 Copyright Printed in USA.

8 Architectural Model SCSI Applications (File Systems, Databases) SCSI Device-Type Commands SCSI Block Commands SCSI Stream Commands Other SCSI Commands SCSI Generic Commands SCSI Commands, Data, and Status SCSI Transport Protocols Parallel SCSI Transport FCP SCSI over FC SCSI over TCP/IP Layer 3 Network Transport TCP IP Layer 2 Network Parallel SCSI Interfaces Fibre Channel Ethernet, PPP, HDLC, 15 Packet Format Preamble Destination Address Source Address Well-Known Ports: 21 FTP 23 Telnet 25 SMTP 80 http 3260 Source Port Sequence Number TCP Header Destination Port Acknowledgment Number Offset Reserved U A P R S F Checksum Options and Padding Window Urgent Pointer bytes Type IP TCP Data Opcode Encapsulated FCS 4 Octet Opcode Specific Fields Length of Data (after 40Byte Header) LUN or Opcode-Specific Fields Initiator Task Tag Opcode Specific Fields Data Field 16 Copyright Printed in USA.

9 for Storage Consolidation IP access to open systems and Fibre Channel storage driver is loaded onto hosts on Ethernet network Able to consolidate servers via onto existing storage arrays Able to build Ethernet-based SANs using arrays Storage assigned on a LUN-by- LUN basis at router -Enabled Hosts (Initiators) Array (Target) Logical Unit Number (LUN): A Field within SCSI Containing up to 64 Bits that Identifies the Logically Addressable Unit within a Target SCSI Device IP Network Router FC Fabric Storage Pool (Target) FC HBA Attached Host (Initiator) 17 for Remote Block Access Block access to remote storage over IP Application must tolerate latency for long distances Metro Ethernet services offer lower latency transport alternative Remote backup of block devices over IP WAN Centralized management from centralized storage Site B Remote Mirrors Storage Pool -Enabled Host Site A FC Fabric IP WAN Router 18 Copyright Printed in USA.

10 Specification Components is an end-to-end protocol has human readable SCSI device (node) naming includes the following base components: IPSec connectivity security Authentication for access configuration Discovery of nodes Process for remote boot MIB standards 19 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 20 Copyright Printed in USA.

11 Naming Initiator and Target require names Name is location independent node name = SCSI device name of device Associated with nodes, not adapters Up to 255 byte displayable/human readable string (UTF-8 encoding) Use SLP, or isns, or query target for names (SendTargets) Two Name Types: iqn Qualified Name eui Extended Unique Identifier (IEEE EUI-64) 21 Name Structure Name Structure Type Unique String iqn Type Organization Date Naming Authority Subgroup Naming Authority or String Defined by Organization Naming Authority iqn com.cisco.1234abcdef da232.betty iqn com.acme.storage.tape.sys1.xyz Date = yyyy-mm when Domain Acquired Reversed Domain Name eui Type EUI-64 Identifier (ASCII Encoded Hexadecimal) eui a425678d 22 Copyright Printed in USA.

12 SCSI/ Relationship SCSI Device = Node (init or target) SCSI Port = Port (init or target) Network Portal defined by IP Addr + TCP Port Portal Group = single SCSI connection Network Entity ( Client) Initiator Node: iqn com.ajax:os1 Initiator Port iqn com.ajax:os1+[isid=1+5+1] Network Portal HBA/NIC Network Portal Port 3000 IP Network Network Portal Port 3000, 5000 Target Port iqn com.ajax:12579[tag=1] Network Portal HBA/NIC Portal Groups Network Portal Port 3000 Target Port iqn. [Tag=2] Target Node: iqn com.ajax:12579 SCSI Device LUs 23 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 24 Copyright Printed in USA.

13 Sessions has the concept of a session Two Session types: (1) Discovery, and (2) Normal Operation Both session types have various phases/stages 1. Initial login phase Security Authentication Operational Parameter Negotiation 2. Full featured phase Session can handle SCSI commands and data after login is complete 25 Connection Allegiance For SCSI commands that require data transfer, the data phase and status phase must be sent over the same TCP connection used by the command phase Consecutive commands that are part of a SCSI task may use different connections within the session (linked commands) Connection allegiance is strictly per-command and not per task Multiple connections allows the session to be scaled across multiple links/devices 26 Copyright Printed in USA.

14 Session Model An session exists between a single initiator (host) and a single target ( router) An session consists of one or more (TCP) connections Session (TCP) Connection TCP/3260 TCP/3260 TCP/3260 Routing Instance Storage Router 27 Connections and SCSI Phases A SCSI command and its associated data and status phase exchanges must traverse the same TCP connection Linked SCSI commands can traverse separate TCP connections for scalability Session (TCP) Connection 1 SCSI Command (1) (Read) SCSI Data (1) SCSI Status (1) SCSI Command (1) (Write) SCSI Data (1) SCSI Status (1) (TCP) Connection 2 Linked SCSI Commands Routing Instance Storage Router 28 Copyright Printed in USA.

15 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 29 Discovery Small networks Static configuration, initiators and targets SendTargets command makes configuration easier Medium-sized networks Service Location Protocol (SLP multicast discovery) Large-sized networks isns (Internet storage name service) Includes soft zone domains Includes database for ongoing management 30 Copyright Printed in USA.

16 Login Sequence (No Authentication) Initiator (PC1) with Driver Discovery: Contact Target and Negotiate Security and Session Parameters Discovery: Solicit Available Targets Normal Login Login to each Target and Negotiate Security and Session Parameters Single TCP Session Establish TCP Session (SYN, SYN/ACK, ACK sequence) 0x03 Login Command SessionType = discovery; InitiatorName=iqn com.acme..PC1 0x23 Login Response (success) Auth=none; HeaderDigest=none; DataDigest=none; 0x03 Login Command SessionType = discovery; InitiatorName=iqn com.acme..PC1 DataPDULength= ; MaxBurstSize= ; 0x23 Login Response (success) DataPDULength= ; MaxBurstSize= ; 0x04 Text Command SendTargets=All 0x24 Text Response TargetName=iqn com.acme..betty 0x03 Login Command SessionType=normal; InitiatorName=iqn com.acme..PC1; TargetName=iqn com.acme..betty 0x23 Login Response (success) Auth=none; HeaderDigest=none; DataDigest=none; 0x03 Login Command SessionType=normal; InitiatorName=iqn com.acme..PC1; TargetName=iqn com.acme..betty 0x23 Login Response (success) DataPDULength= ; MaxBurstSize= ; TargetAlias=betty; Target (SCSI Router) TCP port 3260 (listen) Block Device Has Already Initialized onto the Fibre Channel Fabric 31 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 32 Copyright Printed in USA.

17 Architecture Software Driver Host Storage Router SCSI/TCP Server SCSI Driver FC HBA Fibre Channel TCP/IP Driver GigE NIC Host Driver IP Network TCP/IP Stack NIC Driver NIC Adapter Applications File System Block Device SCSI Generic Adapter Driver SCSI Adapter Path Conventional SCSI Path 33 + TCP Offload Engine (TOE) Hardware implementation of within NIC Offloads TCP and processing into hardware Relieves host CPU from and TCP processing Two forms of offload: Partial offload: Data-path only Full offload: Control and Data-path Wire-rate performance TCP/IP Stack NIC Driver Applications File System Block Device SCSI Generic TOE Driver TCP/IP Stack Adapter Driver SCSI Adapter 34 Copyright Printed in USA.

18 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 35 What Is Meant by High Availability (HA)? Amount of time something is available for use User is concerned about application availability Application is concerned about data availability Both rely on path availability IP Network IP Storage Network Fibre Channel Fabric User Application Is Available if Application Can Be Accessed; This Is Very Dependent on the Path Being Available Application Data Is Available if Data Can Be Accessed and Also Depends on the Path Being Available Focus Data 36 Copyright Printed in USA.

19 What Can Fail in a Storage Network? Failure causes: Hardware, software, or cable failures Mis-configuration or upgrades Intentional attack Application Storage Controller Host Failure NIC or HBA Failure Link Failure Network Failure (Software, Hardware, Links, etc) Controller or Interface Failure Disk Failure 37 High Availability System Approaches Fault Tolerance Failover Load Balancing Load Balancer Fault- Tolerant Host App App Host Cluster App App Suitable for Apps that Do Not Keep Much State; e.g. Web Servers App App Path 1 Path 2 Path 1 Path 2 Path 1 Path 2 Cont Cont Cont Cont Cont Cont RAID RAID RAID 38 Copyright Printed in USA.

20 HA Approaches Use host multipathing software and define redundant and isolated paths to the storage Additional approaches: Static mapping of primary and secondary pwwn paths Master/slave Clustering model (SN542X only) Storage Routers exchange HA heartbeats through management and HA ports Alternate (slave) assumes SCSIrouter identity of failed (master) unit VRRP (Virtual Router Redundancy Protocol) model (MDS9000 IPS-8 only) VRRP Hello Multicasts exchanged in-band Alternate port assumes identity of failed port Can use combinations of the above according to requirements 39 HA Design Multipathing Multipathing: multiple hardware paths to a single drive (LUN) A totally redundant single-host system A similar configuration for clustered servers Host with Multiple() NICs and Multipathing Software Installed Application Multipathing Driver Multipathing includes EMC-Powerpath, Veritas-DMP, and Compaq-SecurePath, etc Ethernet Switches Router A Redundant Fabrics pwwn a Storage Array with Redundant Controller Ports pwwn b Router B 40 Copyright Printed in USA.

21 HA Multipathing Variations Active/Active: balanced i/o over both paths (implementation specific) Active/Passive: i/o over primary path switches to standby path upon failure Application Multipathing Driver Active Active Multipathing Software Balances i/o over Available Interfaces Application Multipathing Active Driver Passive Primary Path pwwn a pwwn b pwwn a pwwn b LUN Mapped over Multiple Paths Using Different Controller pwwns Multipathing Software Monitors Active Path Standby (Failover) Path 41 HA Clustering (SN542x Only) Masters/slaves can exist on different storage routers within HA cluster Both HA and mgmt interfaces used to carry heartbeat Heartbeat also carries hashed config changes from master to slave in XML Failover triggered by: ge or fc interface failure No heartbeat response Manual intervention HA VLAN mgmt VLAN HA mgmt HA Storage Router A ge2 Cluster pebbles fc1 Master SCSI Router barney mgmt SCSI Router barney Slave Slave SCSI Router fred SCSI Router fred Master ge2 fc1 Cluster pebbles Storage Router B 42 Copyright Printed in USA.

22 HA VRRP (MDS9000 IPS-8 Only) Protects against port, card or switch failure Alternate port, card or switch assumes configuration of failed port/card/switch Maintains same virtual IP and MAC addresses Same storage mappings (target WWPN and LUNs) Both cards/switches must be in same FC fabric and LAN/subnet sessions terminated and reestablished (state not retained) Real Address IP: MAC: 00000f Virtual Address IP: MAC: 00000fabcdef Switch A Storage Array Send/receive Multicast Hello Packets Real Address IP: MAC: 00000f Switch B Assumes Virtual IP and MAC address if Switch A Fails Switch B pwwn a 43 HA Static Mapping Protects against FC port/fabric failure, or storage array controller failure LUNs must be visible over both ports (pwwn a and pwwn b) Controller ports can be active/active or active/passive Secondary path will only take over if primary path fails Virtual Target = abc Primary Access = pwwn a Secondary Access = pwwn b Storage Array pwwn a pwwn b 44 Copyright Printed in USA.

23 HA Combinations: Multipathing + Clustering Multipathing and Clustering can be used together or in isolation SCSIrouter defined for each path SCSIrouter will failover when it recognizes a failure Multipathing will failover for other failures through timeouts Application Multipathing Driver Adjacent Link or SCSIrouter Failure Will Trigger Failover to Standby scsirouter pwwn a Mgmt Link HA Link pwwn b 45 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 46 Copyright Printed in USA.

24 Errors What Can Happen? Noise, interference, failures, etc. can flip bits of frames in transit If not detected, potential exists for writing incorrect data Errors must be detected, and then Correct the error, or Throw the frame away Worst Possible Scenario: Silently Store Incorrect Data 47 Data Integrity in Storage Networks Fibre Channel 32-bit Fibre Channel CRC HBA to HBA FC FC End-to-End Optional CRC32C digest Or use IPSec if end-to-end 32-bit Digest Host to Server Endpoints 16-bit TCP Checksum Host to Server TCP Stack 32-bit CRC IPsec Host to Server Endpoints ECC Memory 32-bit Ethernet CRC Optional Always Present 48 Copyright Printed in USA.

25 Data Integrity in Storage Networks (2) Storage Router: Fibre Channel + 32-bit Digest 32-bit IPSec Digest 16-bit TCP Checksum Optional and FC terminated in intermediate Storage Router (SN542x or MDS9000 IPS-8) ECC (Error Checking and Correcting) memory used within endpoints and intermediate routers and storage routers 32-bit Ethernet CRC ECC Memory 32-bit FC CRC FC L2 Switch Storage Router End-point 49 Selecting CRC32C Digest E.g. Windows 2000 Client digest is optional per target Client can force use on or off ( Always vs. Never ) Or, client can negotiate preferred use with target Prefer On Prefer Off 50 Copyright Printed in USA.

26 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 51 Read Initiator Target IP Network Fibre Channel SAN FC SCSI Read 2kB ITT= TCP Ack Data-in Data-in SCSI Status = good TCP Ack SCSI Read 2kB ITT= FCP Data FCP Data SCSI Status = good One-to-one relationship between and FCP frames Single round trip between Initiator and Target to retrieve data 52 Copyright Printed in USA.

27 Write Initiator Target If R2T Required, then Data out Must Wait IP Network R2T (Ready to Transfer) Data out Data out Fibre Channel SAN SCSI Write 2kB ITT= Transfer Ready FCP Data FCP Data FC Two Round Trips for FCP Write SCSI Status=good SCSI Status=good Typical SCSI write requires two round trips Must wait for Transfer Ready before sending data Initiator and Target may negotiate Initial R2T = no during login data out can follow SCSI write single round trip better write service time over distance 53 Latency and Performance If latency is a factor, want to minimize the round trips per unit of work; e.g. a SCSI Exchange Read: Single round trip between Initiator and Target Write: Two round trips if Initial R2T = yes One round trip if Initial R2T = no This is negotiated during Login Most applications are predominantly read oriented 54 Copyright Printed in USA.

28 Latency and Performance (2) Many applications allow multiple outstanding I/Os Lessens the impact of latency on application performance Tape applications permit only a single outstanding I/O (since tapes must write serially) Tapes have a minimum throughput requirement to prevent stop/start or shoe-shining effect Latency consideration is important with tape 55 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 56 Copyright Printed in USA.

29 Network Boot Typical client (e.g. host server) loads in the following order: 1. Operating system (e.g. Windows 2000) 2. Network 3. client driver How can you load the OS over? Network boot uses PXE (Pre-Boot Execution Environment) capability present in many server BIOS and NICs (part of Intel s Wired for Management (WfM) spec) Network and client bootstrapped first allowing subsequent load of OS 57 Network Boot Boot Sequence 1. BIOS sends DHCP request 2. DHCP server returns: Server s IP address and g/way TFTP server address and ROM extension filename server, target, and LUN 3. BIOS uses TFTP to fetch and execute inbp.com file 4. ROM extension sends DHCP request for Boot String 5. DHCP server returns server, target, and LUN 6. ROM extension intercepts INT13 disk r/w and redirects to server 7. BIOS reads C: drive (through inbp.com to load OS (Windows) 8. BIOS executes Windows OS and loads networks and drivers 9. Windows uses driver to access drives (normal operation) Diskless Win2k Server PXE 2.1 Windows OS 9 Driver 8 Network Driver NIC C: D: E: System BIOS ROM Extension UNDI Driver Fibre Channel Server e.g. SN5428 or MDS9000 IPS-8 Fibre Channel Attached Storage Array DHCP Server TFTP Server 58 Copyright Printed in USA.

30 Application of Network Boot 1RU Servers, Blade Servers, Clusters, etc Scaling of 1RU Servers and Blade Servers Replicated C: Drives on storage arrays Easy swap in/out of servers Server expansion DHCP and TFTP Servers Server failure FC FC Centralized C: Drives with Windows OS and Data Drives 59 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 60 Copyright Printed in USA.

31 Security for IP Storage Many services exist within IP to secure IP storage traffic, many of which are not available in Fibre Channel IPSec: FCIP: use IPSec hardware encryption to encrypt FCIP tunnels across the WAN/MAN standard calls for IPSecsupport; requires hardware acceleration in client; very vendor dependent VLANs: Use VLANs to isolate storage traffic within LAN, consider Private VLANs Access Control Lists (ACLs) Use IP and VLAN ACLs to isolate within a LAN; similar to zoning Use storage -router based ACLs to restrict access to storage Authentication: Use RADIUS/TACACS+ to authenticate initiators, 802.1x for switch ports Firewalls: Firewalls can be used due to static TCP ports 61 Design: Dedicated IP Storage Network Front-side IP Network Isolate IP storage network behind application hosts Minimized potential for bandwidth contention Can use a VLAN of existing Ethernet network Recommend use of dedicated Ethernet interfaces on host for attachment to storage network Catalyst Switches Routers FC Fabric Storage Pool Dedicated IP Storage Network Clients -enabled Hosts FC Attached Hosts with HBAs 62 Copyright Printed in USA.

32 SN542X Security Multiple forms of security to ensure an initiators access appropriate routers lp access control lists (ACLs) allow hosts with specific IP address to access router instance LUN mapping to assign storage to initiators RADIUS Server Mgmt Interface 802.1Q Trunks 802.1Q VLAN trunking to only allow access by hosts in a particular VLAN authentication including RADIUS and Tacacs+ support for initiator authentication to SN542X 63 Access Control Access lists restrict access at the target level to specific hosts (initiators) Hosts will only be presented with permitted targets during discovery/login IP Network No Access for SN542x Storage Router SCSI Router barney target db LUN 0 target LUN 0 LUN 1!! ACCESSLIST! accesslist server1 accesslist server / accesslist server2 accesslist server / !! SCSIROUTER!.. scsirouter barney target db accesslist server1.. scsirouter barney target accesslist server2..! SN542X Configuration Target db Can Only Be Accessed By Hosts In Server1 Accesslist Server1 Addresses Defined in Accesslist 64 Copyright Printed in USA.

33 SCSI Router Addressing (SN542x) Each SCSI router uses a unique IP address hosted on the ge1 or ge2 interfaces IP Network SN542x Storage Router SCSI Router barney target db LUN 0 target LUN 0 LUN 1!! SCSIROUTER!.. scsirouter barney serverif ge / scsirouter fred serverif ge / ! Host servers (initiators) point to SCSI router IP address Access lists determine available targets within each SCSI router 65 Restricting Access with VLANs VLAN 100 VLAN scsirouter fred serverif ge2 vlan /24 scsirouter barney serverif ge2 vlan /24 SN542x scsirouter fred vlan target wilma 802.1Q Trunk Fibre Channel ge1 scsirouter barney Fabric vlan target betty Target wilma Target betty Servers 1 and 2 see only SCSIrouter fred and defined targets wilma Servers 3 and 4 see only SCSIrouter barney and defined targets betty Use access lists to restrict access further 66 Copyright Printed in USA.

34 and Private VLANs Private VLANs isolate servers from each other Each server can reach both routers but not any other host Private VLANs represent a hardwarelevel isolation Private VLANs are available on the Catalyst 6000, 4000, 3550, and 3500 family of switches -Enabled Hosts 67 AAA Authentication Allow ONLY authenticated initiators to access configured LUNs Authentication of initiators by routers; and Authentication of router by initiator (2-way authentication) Choice of authentication methods RADIUS external server TACACS+ external server (SN5428 only) Local username database on storage router CHAP (Challenge Handshake Authentication Protocol) used as authentication method 68 Copyright Printed in USA.

35 Authentication (Cont.) SCSI routing service passes username and MD5-hashed password from initiators to AAA server Configured authentication list determines which method to use (local, RADIUS, or TACACS+) User1/pwd1 User2/pwd2 / RADIUS Server TACACS+ Server (SN5428 only) User1/pwd1 User2/pwd2 / User1/pwd1 User2/pwd2 / Hosts (Initiators) User1/pwd1 CHAP RADIUS TACACS+ Local Authentication Services List AAA Authentication Services IP Network SCSI Routing Instance Storage Router FC Fabric 69 Authentication (Initiator) Detail Initiator (PC1) with Driver Contact Target and Negotiate Security and Session Parameters CHAP Algorithm #5 Selected (MD5) Session Parameters Negotiated Single TCP Session Establish TCP Session (SYN, SYN/ACK, ACK) 0x03 Login Command SessionType = discovery; InitiatorName=iqn com.acme..PC1 0x23 Login Response (success) Auth=CHAP; HeaderDigest=none; DataDigest=none; 0x03 Login Command SessionType = discovery; InitiatorName=iqn com.acme..PC1 CHAP_A=5; DataPDULength= ; MaxBurstSize= ; 0x23 Login Response (success) Initiator Sends Username Plus 128- bit MD5 Hash of 0x03 Login Command [challenge + password] 0x23 Login Response (success) Initiator Solicits for Available Targets CHAP_A=5; CHAP_I=218; CHAP_C=0x (128 bits) CHAP_N=User1; CHAP_I=218; CHAP_R=0x678 abc (128 bits) 0x03 Login Command DataPDUlength= ; MaxBurstSize= ; 0x23 Login Response (success) DataPDUlength= ; MaxBurstSize= ; 0x04 Text Command SendTargets=All 0x24 Text Response TargetName=iqn com.acme..betty Target (SCSI Router) TCP Port 3260 (Listen) Authentication Enabled for SCSIrouter CHAP 128-bit Challenge Sent Response Forwarded to AAA Server Authenticated! Send Login Response Block Device Initialized onto Fibre Channel Fabric RADIUS access request Username=user1; CHAP_R=0x678 RADIUS Access Accept RADIUS Server 70 Copyright Printed in USA.

36 Host Configuration for Authentication e.g. Windows 2000 Client Address of Target scsirouter Username and Password Corresponds to Username/ Password in AAA Server or scsirouter Local Database 71 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 72 Copyright Printed in USA.

37 Design Considerations Know your application I/O profile Know your application throughput Determine needed availability Best performance achieved from local (no latency) dedicated (no competing traffic) IP storage network But if you need to cover distance 73 Performance for IP Storage Many services exist within IP to ensure performance in an IP SAN as well as protect IP storage traffic from potential bottlenecks QoS: : use QoS to protect traffic and prioritize it within the LAN for higher priority queuing and switching FCIP: use QoS to throttle non-ip storage traffic to protect FCIP traffic in a WAN/MAN EtherChannel: Use EtherChannel to bundle up to 16Gbps of bandwidth into one logical link within the LAN 100Mbps Ethernet: Potentially use 100Mbps built-in host NIC for applications with lower performance requirements 74 Copyright Printed in USA.

38 High Availability for IP Storage Many services exist within IP and Ethernet to ensure availability in an IP SAN Spanning Tree Enhancements: Use 802.1w for fast spanning tree re-convergence in Ethernet Use UplinkFast and PortFast on Catalyst switches for fast recove ry from a topology change Routing Protocols: Tune OSPF and EIGRP routing protocols for very fast recovery from a routed IP network link/router failure Host Multipath Support: Use multipath software above the host-based layer to add path resiliency to an IP SAN (Veritas DMP, EMC Powerpath, etc.) Hot-Standby Routing Protocol (HSRP) Use HSRP to create redundant default gateways for initiator hosts to ensure fast recovery if a gateway fails 75 and QoS IP QoS mechanisms have default behaviors that must be properly applied for storage: Congestion Avoidance: Drop frames to pace TCP flow (drops) Scheduling: Service high priority queues (latency) Policing: Drop frames that exceed limit (drops) Shaping: Buffer frames and shape output (latency) Although TCP and SCSI I/O are robust protocols and will retransmit, latency is introduced traffic can easily be classified well-known TCP port number 3260 QoS is used to protect IP storage traffic from other traffic around it apply QoS mechanisms to other traffic 76 Copyright Printed in USA.

39 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 77 Modes of Operation SN5428 router and MDS9000 IPS-8 both map: Virtual targets to physical Fibre Channel targets Virtual Fibre Channel initiators to physical initiators SN5428 operates in: SCSI routing mode Transparent SCSI routing mode MDS9000 IPS-8 operates in: Transparent mode 78 Copyright Printed in USA.

40 Logical Representation IP FC FC Targets Presented as Targets Initiators Presented as FC Initiators iqn.def.123 iqn.ghi.456 pwwn d pwwn a IP FC iqn.abc.123 iqn.jkl.555 Storage View pwwn c pwwn b Fibre Channel SAN View 79 Routing Discovering Storage show devices Output from SN5428 [5428-B]$ show devices Fabric Attached Devices detected Interface WWPN PortId Device Type Lun Lunid Type Lunid fc fa6c 0x6501e1 Disk 0 IEEE Reg Name fa6c fc ed76 0x6501e2 Disk 0 IEEE Reg Name ed76 Lun Description Table Interface WWPN Lun Capacity Vendor Product Serial fc fa6c 0 17GB IBM DDYF-T18350R TEFHT396 fc ed GB IBM DDYF-T18350R TEFHQ078 show fcns database Output from MDS # show fcns data... VSAN 10: FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE x6100e1 NL 50:05:07:62:06:88:e8:2c (IBM) scsi-fcp:target 0x6100e2 NL 50:05:07:62:06:88:ee:7d (IBM) scsi-fcp:target 0x N 10:00:00:0b:be:57:75:01 (Cisco) ipfc 0x N 20:03:00:0b:be:57:75:02 (Cisco) scsi-fcp:init isc..w Details of Fibre Channel attached devices is stored in switch name server database This information can be used to configure the virtual to physical Fibre Channel storage mapping 80 Copyright Printed in USA.

41 Mapping Virtual Targets to Physical Fibre Channel Storage targets map to physical storage devices Two methods possible: Target-and-LUN Mapping Specify individual physical LUNs Target-only Mapping Map all LUNs within the target 81 Target-and-LUN Mapping SN5428 configuration: Map to a uniquely identifiable LUN using one of: WWPN + LUN LUNWWN (not always assigned) Serial Number WWPN Addressing: scsirouter fred target lun 0 wwpn 20:00:07:63:00:c5:87:52 lun 2 LUNWWN Addressing: scsirouter fred target lun 0 lunwwn xx:xx:xx:xx:xx:xx:xx:xx Switch or Storage Router Physical Storage Array WWPN: 20:00:07:63:00:c5:87:52 LUN 0 Serial# Serial Number Addressing: scsirouter fred target lun 0 serial Target LUN 1 Serial# MDS9000 configuration: LUN 0 LUN 2 Serial# iscsi virtual-target name iqn. pwwn 20:00:07:63:00:c5:87:52 fc-lun 2 iscsi-lun 0 LUN 3 Serial# Copyright Printed in USA.

42 Target-Only Mapping Map an target to a physical storage target and all its LUNs using WWPN (pwwn) SN5428 Configuration: WWPN Addressing: scsirouter fred target wwpn 20:00:07:63:00:c5:87:52 Switch or Storage Router Target LUN 0 Physical Storage Array WWPN: 20:00:07:63:00:c5:87:52 LUN 0 Serial# LUN 1 LUN 1 Serial# MDS9000 Configuration: iscsi virtual-target name iqn. pwwn 20:00:07:63:00:c5:87:52 LUN 2 LUN 3 LUN 2 LUN 3 Serial# Serial# Mapping to Host Server Each mapped LUN appears as a local disk volume to the host server Server e.g Win2K Server SCSI Router fred target Target-only Mapping Physical Storage Array WWPN: 20:00:07:63:00:c5:87:52 Loopid: 129 Local Disk (D:) LUN 0 LUN 0 Serial# Local Disk (E:) LUN 1 LUN 1 Serial# Local Disk (F:) LUN 2 LUN 2 Serial# Local Disk (G:) LUN 3 LUN 3 Serial# Copyright Printed in USA.

43 Mapping to Host Server Multiple SCSI Routers Server e.g Win2K Server SN542x Storage Router SCSI Router barney target web Target-and-LUN Mapping Physical Storage Arrays WWPN: 20:00:07:63:00:c5:87:77 LUN 0 Serial# Local Disk (D:) LUN 0 LUN 1 Serial# Local Disk (E:) target db WWPN: 20:00:07:63:00:c5:87:52 Local Disk (F:) LUN 0 Target-and-LUN Mapping LUN 0 Serial# Local Disk (G:) SCSI Router fred LUN 1 Serial# target LUN 0 Target-only Mapping WWPN: 20:00:07:63:00:c5:87:52 LUN 0 Serial# LUN 1 LUN 1 Serial# Mapping Hosts to Virtual FC Hosts hosts can be (1) Dynamically or (2) Statically mapped Dynamic: N_Port, nwwn and pwwn allocated from pool each time host connects Static: nwwn and pwwn mapping is kept permanent (automatically from pool or user selected) Used with storage arrays that use access control and LUN masking based upon host pwwn and nwwn Host george IP FC george Presented to Fibre Channel Fabric on Virtual N_Port 86 Copyright Printed in USA.

44 IP Storage Network Design Storage Networking Technology Review and IP Storage Networking Naming TCP and Sessions Discovering Storage Connecting Servers NICs and TOEs High Availability Data Integrity Reading and Writing with Network Boot IP Storage Network Security Designing the IP Storage Network Configuring IP Storage How Much Bandwidth? Profiling Applications 87 Determining the I/O Profile Windows NT/2000 perfmon Write to.csv for later analysis Select disk bytes read/written per second Monitor CPU load Linux and UNIX iostat -D option shows I/O per sec Use scripting to monitor over time Monitor CPU load with ps or top iostat -D /dev/dsk/c0t0d0s7 sd15 sd16 sd36 sd75 rps wps util rps wps util rps wps util rps wps util Copyright Printed in USA.

45 Microsoft Exchange Observations Microsoft loadsim generator Moderate I/O rate independent of storage size Negligible difference between FC HBA and software driver performance TOE will reduce CPU% Server becomes scaling point rather than storage 100 CPU% 60 MB/s 50 Server CPU Impact of vs. HBA GigE NIC s/w driver 6 Users FC HBA I/O Throughput Users FC 89 Microsoft SQL Server Observations Moderate I/O rate 10/100 NIC with adequate for most apps but dependent upon app Negligible difference between FC HBA and software driver performance Use TOE if CPU % is a concern 100 CPU% MB/s Server CPU Impact of vs. HBA GigE NIC s/w driver FC HBA Users I/O Rate and Throughput Users MB/s I/O per Sec FC I/O sec 90 Copyright Printed in USA.

46 Recommended Reading Designing Storage Area Networks, Second Ed. ISBN: Essential Guide to Storage Area Networks ISBN: Available on-site at the Cisco Company Store 91 Please Complete Your Evaluation Form Session 92 Copyright Printed in USA.

47 93 Copyright Printed in USA.