Deploying FC and FCoE SAN with IBM System Storage SVC and IBM Storwize platforms

Transcription

1 Deploying FC and FCoE SAN with IBM System Storage SVC and IBM Storwize platforms Guide v1.0 Bhushan Gavankar, Subhadip Das, Aakanksha Mathur IBM Systems and Technology Group ISV Enablement August 2014 Copyright IBM Corporation, 2014

2 Table of contents Abstract...1 Getting started with FC and FCoE technology...1 Deploying FC and FCoE...2 Initiator (host) side settings... 2 For Red Hat Enterprise Linux (RHEL) or SUSE Linux Enterprise Server (SLES) operating systems...3 For VMware ESX...3 Microsoft Windows 2008 R Switch settings... 7 Understanding the traffic...7 Creating the zones...9 Planning for separate fabrics...9 Priority-based flow control in FCoE SAN...10 Target side settings Creating host objects using the host port WWPNs that can be seen by SVC and Storwize Assigning VDisks to each host object...12 Quick recap...15 Summary...15 Resources...16 About the authors...16 Trademarks and special notices...17

3 Abstract The intent of this paper is to provide guidelines that can help customers (storage administrators) to quickly deploy Fibre Channel (FC) and Fibre Channel over Ethernet (FCoE) storage area network (SAN) with IBM System Storage SAN Volume Controller (SVC) and IBM Storwize platforms. The paper takes in account the different I/O characteristics and hardware options available in the platforms, while making recommendations, which should lead in getting optimal performance for application workloads. FC and FCoE are standard SAN transport protocols deployed in many data centers and hence in-depth explanation of the protocols is not the objective of this paper. This paper briefly introduces both the protocols and lists down the key points that can aid in successful deployment. The primary focus of this paper is to assist storage administrators in understanding various configurations required on storage elements, that is, hosts (initiators), switches that can either be FC or Fibre Channel Forwarder (FCF), and storage arrays (targets) in order to deploy FC and FCoE SAN, using SVC and Storwize environments to achieve optimal performance and fault tolerance. Getting started with FC and FCoE technology FC and FCoE are both technologies used to transport Small Computer System Interface (SCSI) command descriptor blocks (CDB) between storage initiators (application servers) and storage targets (IBM System Storage SAN Volume Controller or IBM Storwize ). So, in essence, both FC and FCoE are commonly referred as transport protocols in SAN. Note: SVC and Storwize perform the role of an initiator when virtualizing storage arrays connected to them. Figure 1: FC and FCoE protocol layers As shown in Figure 1, the top three layers of FC were retained, as is, in FCoE implementation. However, the lower two layers, namely FC-0 and FC-1 in FC, were replaced with the Ethernet layers, namely physical and Media Access Control (MAC). As such, all functions implemented in FC2 and above layers are same across FC and FCoE SAN, that is, fabric login (FLOGI), port login (PLOGI), process login (PrLOGI), and error recovery. This means, worldwide name (WWN) is there in both the protocols and hence the requirement for switch zoning is common for SANs based on any of these protocols. The difference is the physical medium characteristics and bit encoding used in the protocols. While FC (till 8 Gbps) uses 8b/10b encoding giving a protocol efficiency of 80%, FCoE uses more advanced 64b/66b encoding scheme allowing 96% protocol efficiency. 1

4 Deploying FC and FCoE Any SAN fabric will have the following three key elements: Initiator: This is any device which initiates an I/O request to storage array over I/O interconnect. Target: The end point that receives I/O commands and acts on the I/O requests from an initiator. I/O interconnect: Any path used to transfer data and control information between components of a SAN fabric. This includes the cables, transreceivers, and the switches that make up the path between the initiator and the target. In order to deploy an FC or FCoE SAN, storage administrator would need to do some configuration on all three elements, namely initiator, target and switch. However, the level of configuration and complexity will vary significantly across the three elements. The following sections provide more details on the configurations that would be required on the three elements when using SVC or Storwize systems. Initiator (host) side settings As far as FC and FCoE SANs are concerned, the configuration required on the initiators is minimum. Typically, the following steps should suffice: 1. Insert a supported host bus adapter (HBA), that is, FC HBA in case of FC SAN or converged network adapter (CNA) in case of FCoE SAN. 2. Install a supported driver for the HBA. 3. Note down the worldwide port names (WWPN) of the HBA installed in each host (initiator). This information will be used when creating zones on FC switch or FCF switch and also for mapping required VDisks, logical unit numbers (LUN), from SVC and Storwize to the initiators. 2

5 For Red Hat Enterprise Linux (RHEL) or SUSE Linux Enterprise Server (SLES) operating systems Figure 2: Checking WWPN in Linux OS For VMware ESX Using the graphical user interface (GUI) 1. Go to the VMware vsphere Client or vcenter 2. Connect to the ESX Host using its management IP address 3. Select the host on the left panel (as shown in the following figure) 4. Click the Configuration tab (at the right pane). Notice that all the HBAs are listed for the host with their worldwide IDs (WWIDs). 5. Get the worldwide node name (WWNN) or WWPN for your HBA 3

6 Figure 3: Checking WWPN in ESX through GUI Using command-line interface (CLI): Go to the management console of ESX and run the following command: ~#esxcli storage core adapter list This shows you the WWPN for the attached HBAs. Figure 4: Checking WWPN in ESX through CLI 4

7 Microsoft Windows 2008 R2 Microsoft Windows 2008 R2 has an in-build storage explorer feature. Using this, a storage administrator can easily get the WWPN for the attached HBA without requiring any third-party tool installed. 4. In the Run prompt, type mmc. Figure 5: Open Microsoft Management Console (MMC) in Windows OS 5. A console window opens. Click File Add/Remove Snap-in. Figure 6: Adding snap-in in MMC 5

8 In the new window that opens, select Storage Explorer and click Add to move it to the right pane. Note: In some cases a pop-up might ask you to select either a local computer or a remote computer. Select Local computer. Figure 7: Add Storage Explorer in MMC 6. After Storage Explorer is selected and added to the right pane, click OK. After that, a new window of Storage Explorer opens with all the information about the attached HBAs, including the WWPNs. 6

9 Figure 8: WWPN in Storage Explorer in Windows 8 R2 Switch settings The configuration of switch is generally very time consuming and also the most-complex part. Any mistake here can result in effects that can vary between any one or combination of (1) initiators not seeing VDisks, (2) suboptimal performance, and (3) loss of connectivity. A thorough planning is required before deciding the zoning strategy and extreme care while doing actual configuration is required to alleviate the problems typically seen during SAN deployments. Understanding the traffic It is important to understand the data traffic that flows between different elements and design the SAN fabric accordingly. In case of SVC, the overall traffic passing through the SVC nodes (I/O ports) can be divided into the following three categories: 7

10 Initiator to SVC traffic: This is the I/O traffic between SVC nodes and hosts (application servers). This is the most important traffic and performance seen by applications and users is closely related to this traffic. Clustering traffic: This is the I/O traffic between SVC nodes, forming a cluster. Typically when host I/O traffic is composed of only Read workloads, SVC clustering traffic is minimum and consumes minimal SVC resources (processor, port bandwidth). However, when host I/O traffic is composed of Write workloads, due to cache mirroring, a significant portion of SVC resources, especially bandwidth, is taken up by SVC clustering traffic. In order to get good performance for host I/O, we should try to keep SVC clustering traffic and host I/O traffic on separate I/O ports, that is, SAN fabrics. SVC clustering is currently supported only over FC, FCoE, or both fabrics. Note: No inter-switch link (ISL) hops are permitted among SVC nodes within the same I/O group, and maximum three ISL hops are permitted among SVC nodes, provided they are in different I/O groups. However, for optimal performance and reliability, it is recommended that SVC clustering and back-end traffic does not pass over ISLs. SVC to back-end storage traffic: This is the I/O traffic between SVC nodes and backend storage which is virtualize by SVC. Depending on the host I/O workloads, back-end storage traffic can vary from medium to heavy. Preferably, you should try to keep backend traffic and host I/O traffic on separate SAN fabrics. Back-end traffic is currently supported only over FC, FCoE, or both fabrics. SVC 2145-CG8 and SVC 2145-DH8, both support having 8Gbps FC ports and 10Gbps CNA that can carry FCoE and 10G Internet Small Computer System Interface (iscsi) protocol. If your SVC node s hardware configuration has two adapters, which can be either same type (that is, only 8G FC or 10G CNA) or mixed (that is, one 8G FC and one 10G CNA), then it is recommended to have two separate fabrics; one for the host I/O traffic and another for SVC clustering, plus back-end traffic. In scenarios that use a single Storwize control enclosure to form a cluster, the most likely traffic passing through the node ports is only host I/O. This is because the clustering traffic between Storwize canisters can pass over the internal link between the two canisters. Also, traffic to disks passes over the internal serial-attached SCSI (SAS) links. Hence, from a switch zoning perspective, you only need to create separate zone for each host / host port to Storwize canister port. In order to get optimal performance, it is recommended not to have ports from different Storwize canisters in same zone. For deployments that have more than one Storwize control enclosures in a single cluster, node ports would also need to carry clustering traffic. As cache mirroring happens between nodes within the same I/O group, the clustering traffic passing through node ports of Storwize control enclosures should be minimum. However, where possible, the recommendation of separating out host I/O traffic and clustering traffic on different fabrics, should be implemented. Also, in case of configurations that deploy an external storage behind Storwize control enclosure, you might need to create zones for back-end storage traffic, as described in the next section. Note: For detailed guidelines on creating zones, refer to IBM Knowledge Center for IBM Storwize V7000 at: ibm.com/support/knowledgecenter/st3fr7_7.2.0/com.ibm.storwize.v doc/svc_configrulessumm ary_ html?lang=en 8

11 Creating the zones The following guidelines of creating zones are recommended Zones for host I/O traffic: It is recommended that you create separate zone for each host / host port and SVC node ports. Do not mix SVC node ports from different nodes in same zone, that is, ensure that SVC node ports from different nodes cannot see each other in zones created for host I/O traffic. Zones for SVC clustering traffic (not required for Storwize): This zone should only have SVC node ports from all nodes which would be part of single cluster. This zone should not have any host ports, back-end storage ports, or both. Zones for back-end storage traffic (not required for Storwize): It is recommended that you create separate zones for each SVC node and back-end storage system ports. Ensure you do not have SVC node ports from different SVC nodes in same zone. Also, ensure you do not have ports from different back-end storage systems in same zone. Note: For detailed guidelines on creating zones, refer to IBM Knowledge Center for SAN Volume Controller (2145) at: ibm.com/support/knowledgecenter/stpvgu_7.2.0/com.ibm.storage.svc.console.720.doc/svc_configrules summary_ html?lang=en Planning for separate fabrics SVC 2145-CG8 and SVC 2145-DH8, both support having 8Gbps FC ports and 10Gbps CNA that can carry FCoE and 10G iscsi protocol. If your SVC node s hardware configuration has both the adapters, that is: 8Gbps FC and 10Gbps CNA, then its recommended to have two separate fabrics; one for host I/O traffic and another for SVC clustering, plus back-end traffic. Note: 8Gbps FC protocol uses 8b/10b encoding scheme, which gives a protocol efficiency of 80%; whereas, 10Gbps Ethernet uses 64b/644b encoding scheme, which improves the protocol efficiency to around 96%. For optimal performance, it is recommended to select fabric with larger available bandwidth for carrying host I/O traffic and keep the fabric with lesser bandwidth reserved for SVC clustering and SVC to back-end storage traffic. SVC 2145-CG8 will have four 8Gbps FC ports in base model giving a maximum bandwidth of 32 Gb with FC fabric and around 25 Gb usable bandwidth for user data. In addition, customers can optionally purchase a dual port 10 Gbps CNA that can give a maximum bandwidth of 20 Gb and around 19 Gb of usable bandwidth for user data. SVC 2145-DH8, flexibility provided in terms of various adapters supported in DH8, makes the decision a bit more complex. DH8 will have three 1Gbps Ethernet ports in base model. However, it can support a maximum of one quad port CNA adapter or up to three 8Gbps FC adapters, or a combination of both, if the cards are inserted in the three valid Peripheral Component Interconnect Express (PCIe) slots. Note: Refer to IBM Knowledge Center for detailed guidelines on the possible hardware configurations. 9

12 Priority-based flow control in FCoE SAN Traditional Ethernet is unreliable and does not offer any guarantee that packets sent by source will be received by destination. Reliability is achieved by means of upper layer protocols. Many switch vendors support priority-based flow control (PFC), which extends the basic PAUSE frame concept across various levels of class of service (CoS). This enables an administrator to assign different priority for a certain type of traffic over other traffic. Hence, at times of congestion, high important traffic can continue to flow while non-important traffic CoS will be paused. A detailed discussion on PFC is out of context of this paper, however, storage administrators needs to be cognizant of this feature in switches and ensure that it has been set up properly in order to get acceptable performance. It is recommended to read the relevant switch vendor documentation to configure the switches properly. For Cisco switches, you can refer to the Priority Flow Control: Build Reliable Layer 2 Infrastructure white paper on PFC for more details at. Target side settings The GUI steps or CLI commands used with SVC and Storwize are the same and therefore, the following information applies to both the platforms. This section provides the three basic steps that must be run on the target side. These three tasks can be run either by using CLI commands or by GUI. Noting the WWPNs of node ports, with which zones can be created In order to figure out the WWPNs of node ports, you can use the CLI command, svcinfo lsportfc. The output of this command is shown in the following figure. Figure 9: Output of lsportfc command as seen on a SVC DH8 cluster 10

13 In the configuration shown in Figure 9, each DH8 node in cluster has been configured with one quad port 8Gbps FC adapter and one quad port 10Gbps CNA. Hence, you can see total of 8 FC 8Gb ports and 8 Ethernet 10Gb ports across I/O group. Note: Storwize V7000 model 524 also shows similar output, provided each of the canisters has one quad port 8Gbps FC adapter and one quad port 10Gbps CNA. Apart from the WWPN column that gives the WWPN of each port that you want to note down before proceeding with zoning on the switch, there are few other fields that are very useful while troubleshooting basic connectivity issues. The status of each port is shown to be active; which means the port is connected to a switch port and the link is up. A status of inactive means that the port is not able to establish a connection with the switch. Possible reasons would be bad or lose cable, bad transreceiver, faulty switch port, and so on. Figure 10: Output of lsportfc command, as seen on a SVC CG8 cluster In the configuration shown in Figure 10, each CG8 node in cluster has been configured with one quad port 8Gbps FC adapter and one dual port 10Gbps CNA. Hence, you can see a total of eight FC 8 Gb ports and four Ethernet 10 Gb ports across I/O group. Note: Storwize V7000 Gen1 also shows similar output, provided each of the canisters has one quad port 8Gbps FC adapter and one dual port 10Gbps CNA. Creating host objects using the host port WWPNs that can be seen by SVC and Storwize Again, you can get this task accomplished using CLI or GUI. Creating host objects, using CLI Syntax of the typical CLI command to create a host object using WWPN is: svctask mkhost -name hostone -fcwwpn E08B251DD4:210100F08C262DD8 -force -type rhel -iogrp 1:iogroup3 In this example, a command is given to create a host object called hostone, with FC WWPNs E08B251DD4:210100F08C262DD8. The test team has opted for force creation and bypass validation of WWPN by using the force flag. The host is running the RHEL operating system. The host can access the VDisk from I/O group with ID of 1 and name iogroup3. 11

14 Creating host objects using GUI 1. You need to launch the web browser with the cluster IP address and click Hosts (fourth icon from top on the extreme left side of the browser 2. Click Create Host at the upper-left side and for selecting FC or FCoE host, select a Fibre Channel host type in the pop-up window. 3. In the dialog box (as shown in Figure 11) select the WWPN and enter a host name. Note that the text file that you created listing WWPNs in each host, might be helpful now. You can combine WWPNs that are physically in the same host, under a single host object. If multipath is properly set up on the host, then it should take care of combining raw devices seen through separate paths in one single multipath disk (dm device) that might be exposed to applications. Figure 11: Select WWPN and create host Assigning VDisks to each host object Assuming that you have already created the VDisks, the only task that is left on SVC or Storwize side is to map these VDisks to the host objects that were created in the previous step. 12

15 A very simple CLI command to do this job is: svctask mkvdiskhostmap -host host1 vdisk5 As shown in this example, you are mapping virtual disk vdisk5 to host1. While there are multiple ways to do host mappings from GUI, an easier option might be to do the mappings after creating the host objects. Figure 12: Select host to assign VDisk 1. Right-click the host object and click Modify Mappings. 13

16 Figure 13: Select VDisk to be mapped to selected host 2. In the Modify Host Mappings dialog box, select the appropriate host object to which the VDisks should be mapped. Select the VDisks shown in the Unmapped Volumes section and using the right arrow button, move them to the Volumes Mapped to the Host section. 3. Click Map Volume or Apply to make the changes in mapping. Note: There are many ways to perform the mapping. To understand other ways, refer to IBM Knowledge Center. On completion of all above steps, your initiators should see the VDisks and start using them for application I/O. Note: SVC and Storwize 10G ports support handling iscsi traffic and FCoE traffic simultaneously. However, both these protocols share the same hardware resources from the HBA. In a setup where iscsi and FCoE IO is happening simultaneously, additional precautions must be taken when designing the SAN and during implementation to ensure that the applications get the expected performance. 14

17 Quick recap The following points can act as a quick checklist during planning or implementing phase. Install HBA and HBA driver in every host (ensure you are following IBM support matrix). Connect the appropriate transreceivers and cables between HBA ports to FC or FCF switch ports and from FC or FCF switch ports to SVC or Storwize node ports (account for balancing I/O across paths during laying cables). List down the WWPN being used in each host. Ensure that appropriate multipathing drivers are installed and multipath configuration is accurate. List down the SVC and Storwize WWPNs as seen through the lsportfc command. On switch, carefully plan and run zoning. Keep in mind the recommendations provided in this paper, while creating your zoning plan. On SVC and Storwize, create host objects and then complete VDisk mapping to each host object. From host, verify that you are seeing all the VDisks that were mapped to the host object in the previous step. Summary The paper explains various I/O characteristics and hardware options, which should be considered during design and implementation of FC and/or FCoE SAN environments. This paper provides guidelines which can enable storage administrators to create high performant and fault tolerant SAN using IBM SVC and IBM Storwize platforms 15

18 Resources The following websites provide useful references to supplement the information contained in this paper: IBM Systems on IBM PartnerWorld ibm.com/partnerworld/systems/ IBM Knowledge Center ibm.com/support/knowledgecenter/?lang=en IBM Publications Center IBM Redbooks ibm.com/redbooks IBM developerworks ibm.com/developerworks IBM Whitepaper on SVC DH8 and v7000 Gen2 ibm.com/partnerworld/page/stg_ast_sto_wp_introduction-to-ibm-system-storage-svc dh8/lc=en_all_zz About the authors Bhushan Gavankar is a test architect in IBM Systems and Technology Group SVC and Storwize family with primary focus around performance testing across iscsi and FCoE protocols. He has around 15 years of experience in networking, systems, high availability and storage domains. You can reach Bhushan at bgavanka@in.ibm.com. Subhadip Das is a test engineer in IBM Systems and Technology Group SVC and Storwize family with primary focus around performance testing across FC, iscsi and FCoE protocols. He has around 6 years of experience in systems, virtualization, clustering and storage domains. You can reach Subhadip at subhadid@in.ibm.com. Aakanksha Mathur is a test engineer in IBM Systems and Technology Group SVC and Storwize family with primary focus around performance testing across iscsi and FC protocols. She has around 5 years of experience in servers, virtualization and storage domains. You can reach Aakanksha at aamathur@in.ibm.com. 16

19 Trademarks and special notices Copyright IBM Corporation References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the 17

20 storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 18