EqualLogic Storage and Non-Stacking Switches. Sizing and Configuration

Transcription

1 EqualLogic Storage and Non-Stacking Switches Sizing and Configuration

2 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. All other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell, Inc. is strictly forbidden. For more information, contact Dell. August 08 Page 2

3 Introduction With the introduction of the Dell EqualLogic PS Series of Storage Area Network (SAN) arrays and the PowerVault MD3000i, Internet SCSI (iscsi) has been promoted to a prominent role in the storage strategy for Dell. With the focus on iscsi as a viable solution for Enterprise SAN solutions, several questions have arisen with respect to sizing and configuring Ethernet network infrastructures to help support this new SAN technology. This whitepaper will focus on answering these questions, for the PS Series array and non-stacking switch offerings such as the PowerConnect 54xx series. While we focus on the PowerConnect switch as an example of a non-stacking switch, the guidelines and sizing exercises presented in this whitepaper hold true for any non-stacking switch (commonly referred to as a Layer 2 switch). EqualLogic storage solutions offer a fundamental change in iscsi SAN implementation. By configuring PS Series storage arrays into a SAN group, both large and small businesses can incrementally deploy a modular and affordable iscsi SAN solution that easily scales to hundreds of terabytes. Built on a ground-breaking, patented architecture, EqualLogic arrays deliver enterprise-quality performance and reliability for all major operating systems. Intelligent automation and seamless virtualization of a single pool of storage greatly simplify storage management. EqualLogic arrays also provide a comprehensive set of high-value features, in addition to fully redundant, hot-swappable hardware. The Dell PowerConnect 5400 Series delivers wire-speed, gigabit Ethernet with advanced security and Enterprise management features in a non-stacking switch to help meet the needs of small to medium size businesses. In addition to offering a secure, high-performance networking platform and delivering advanced network management features, the PowerConnect 5424 and PowerConnect 5448 are also capable of self-optimization for iscsi storage traffic and make an excellent choice for small to medium iscsi storage solutions based on the PS Series storage arrays. Properly Sizing an iscsi Switch Environment to Support the PS Series Arrays One of the most important steps in planning for an iscsi SAN is properly determining the size of the SAN solution required then sizing the network infrastructure appropriately to allow for current and future growth. These assessments should focus on several factors including storage capacity and performance requirements. The first criterion is measured in bytes (typically gigabytes and terabytes), and the second criterion is measured in the number of operations per second (IO/s) you require for all of your hosted applications that will use the SAN. While most people think of the amount of storage as a simple question to answer give me the amount of storage I need at the lowest cost it is not quite so simple. The actual question that should be asked and answered in a sizing exercise such as this is What is the number of IOs that I need to meet my application performance requirements with the amount of storage that I need? The answer to this question will tell you how many EqualLogic arrays you will need to meet your application requirements, both in size and performance (IO/s). August 08 Page 3

4 A second concept that needs to be understood in sizing a switch infrastructure to support a PS Series array SAN is the concept of inter-switch links (ISLs). A fully redundant EqualLogic SAN will require two switches that are linked together using ISLs. These links are used by the PS Series SAN for scalability when there are multiple arrays in the PS Series SAN to allow a free flow of operational information and application data between arrays. Redundancy is required in any highly available, enterprise-class solution, and we use ISLs between two or more switches to provide this redundancy. In any switch that does not have dedicated inter-switch linking technology, such as the PowerConnect 5400 Series, ISLs must be implemented using one or more of the standard Ethernet ports in each switch. Using multiple ISLs for performance scalability is important for multiple-array SANs since these ISLs will be used to provide additional bandwidth for the SAN arrays to communicate with minimal delay. In an EqualLogic SAN, overall SAN performance is increased when multiple arrays are added to the SAN group and adequate inter-switch bandwidth is provided to allow the arrays to communicate with each other. To gain this scalability, the switch should support combining multiple ISL links into a single, logical link so that the switch can take full advantage of the total bandwidth created in this aggregated link. Putting the concept of array performance (in IO/s) together with the total size requirements, will help you determine the number of EqualLogic arrays you will need, completing the first step in properly sizing the iscsi switch environment. Later steps will take into account the number of arrays and the required amount of bandwidth required to support inter-array communication and load balancing through the use of ISLs. In general, here are the steps in sizing a non-stacking switch infrastructure for a EqualLogic SAN given the number of arrays to meet your storage needs along with the number of iscsi host servers that will be accessing these arrays for a fully redundant infrastructure: 1. Determine the number of EqualLogic arrays that will be required to meet your specific performance and capacity needs. 2. Determine the number of ports required by the SAN group for data movement: Since each EqualLogic array has 3 active data ports and 3 standby data ports, each array requires 6 Ethernet ports for a fully redundant solution. Therefore, multiply the number of EqualLogic arrays in the SAN by Determine the number of hosts that will access the SAN: Multiply this number by 2 (for redundant host connections) 4. Determine the number of ISLs required to support the SAN infrastructure: For each active port on each array count one ISL. Each ISL takes up one port on each switch For example: One array 3 ISLs (for a total of 6 ports) August 08 Page 4

5 0 1 X2 PowerEdge ACT ERR PWR ACT ERR PWR Invalid Address ID X2 0 1 PowerEdge 2950 X ACT ERR PWR ACT ERR Invalid Address ID X2 EqualLogic Storage and Non-stacking Switches Two arrays (6 active data ports) 6 ISLs (for a total of 12 ports) 5. Add the results of steps Divide this result from Step 5 above by 2 (a redundant switch infrastructure will have two switches connected by one or more ISLs). The result is the number of ports you will need on each switch to support a two switch, fully redundant, highest performance EqualLogic SAN infrastructure. For the more mathematically inclined, here is the concise formula: #Arrays x 6 + # hosts x 2 + ( #Active PSArray Ports x2) 2 An Example SERIAL PORT 0 ETHERNET 2 ETHERNET 1 ETHERNET 0 SERIAL PORT 0 ETHERNET 2 ETHERNET 1 ETHERNET 0 SERIAL PORT 0 ETHERNET 2 ETHERNET 1 ETHERNET 0 SERIAL PORT 0 ETHERNET 2 ETHERNET 1 ETHERNET 0 PWR... Host #1 Host #10 6x 1Gb LAG Standby Links Active Links Inter-switch Links Figure 1 So, an example is probably appropriate here. Let s consider a SAN with 2 PS5000 arrays and 10 hosts as illustrated in Figure 1 above. Following the steps listed above, here are the results: 1. Number of EqualLogic arrays for solution: 2 2. Number of ports needed to support two EqualLogic arrays: 2 x 6 = 12 August 08 Page 5

6 3. Number of ports needed to support 10 hosts with redundant Ethernet connections to the SAN: 10 x 2 = Number of ports required for ISLs: we have two EqualLogic arrays, each with 3 active ports giving us 6 total active ports. For each active port, we need one ISL. Each ISL requires 2 ports. So, 2 x 3 x 2 = Adding the results from steps 2 4: =44 total switch ports 6. Dividing this total by 2 to determine the number of ports on each of two switches give us 22 ports per switch. Now, you might be saying, Hey, I can easily use two PowerConnect 5424 switches for this SAN, and you would technically be correct, but you have to be sure that your environment is not going to grow any larger. Understanding that you are only using 22 of the 24 ports on each switch, you also have to understand that you only have 2 ports left on each switch, or 4 total free ports with which to expand your SAN. This means you could add 2 more redundant hosts to the SAN, OR you would only be able to add one more EqualLogic array to the SAN. This is because the addition of a third array would require an additional 12 ports (6 ports for the array connections and 6 ports for the addition of 3 new ISLs). A more flexible solution would be to use PowerConnect 5448 switches. The additional ports could be used for a variety of things; either to expand the SAN or configure additional VLANs for other networking purposes. The following table lists the number of arrays versus the number of hosts that a PowerConnect 5424 and PowerConnect 5448 can support when running a fully redundant infrastructure: 1 Array 2 Arrays 3 Arrays* PowerConnect Hosts 12 Hosts 6 Hosts PowerConnect Hosts 36 Hosts 30 Hosts * Maximum Link Aggregation Group size is 8 ports. Inter-Switch link will now be oversubscribed 3 arrays would require 9Gb of bandwidth, but only 8Gb is available in a single LAG. Table 1 Because there is a limit on the number of ports we can place in a Link Aggregation Group (see Port Aggregation below), the number of EqualLogic arrays in the SAN is limited regardless of the number of available ports on the two switches since a EqualLogic SAN requires an Inter-Switch Link that is not oversubscribed. If a SAN with more than 3 EqualLogic arrays is required, it is recommended that stackable switches such as the Cisco Systems 3750 family or other models of stacking switches be utilized. But, if August 08 Page 6

7 the SAN infrastructure is not expected to grow beyond 3 PS Arrays and 30 hosts, leveraging an Enterprise class non-stacking switch such as the PowerConnect 54xx Series of switches will definitely meet and exceed the requirements for this class of storage solution. PowerConnect 5400 Series Switch Features and the EqualLogic Storage While any Ethernet switch can host iscsi SAN data, some switches perform better than others. The PowerConnect 5424 and 5448 are Enterprise class, non-stacking switches that provide several features that are required to support a robust and high-performance iscsi infrastructure. These features include support for: Virtual Local Area Networks (VLANs), Quality of Service (QoS), Flow Control, Storm Management, Low latency port forwarding modes, and Jumbo Frames. In the next several sections, we will discuss the importance of each of these features and provide links to the online documentation to configure your PowerConnect 5400 Series switch to provide the best performance for an iscsi switch infrastructure. Virtual Local Area Networks Virtual Local Area Network (VLAN) functionality gives a switch, like the PowerConnect 5400 Series, an added level of flexibility to your network infrastructure. By enabling VLAN functionality, you gain the ability to create multiple, virtual networks on a single physical switch. Once configured, each VLAN can be independently configured to meet the special needs of the specific application for which it was created. This is especially important for iscsi SAN implementations that might need to share the physical switch resources with other types of Ethernet traffic, such as the corporate LAN. Traffic isolation is very important for a number of reasons including security and performance. Instructions on implementing Virtual LANs can be found at Dell s support site by using the following link: Configuring VLANs on PC54xx Series Switches Quality of Service Quality of Service (QoS) is a feature that allows the switch administrator to give some Ethernet traffic higher priority than other types of traffic allowing for better performance and predictability when transmitting data. For iscsi, it is very important that the data being transmitted between the host(s) and the SAN arrive in a very short timeframe. For dedicated iscsi switches, QoS does not play a role since all of the traffic on the dedicated infrastructure is storage related, but when the situation arises where the physical LAN is being used for both corporate and iscsi traffic, while not a recommended configuration, a switch that is properly configured with QoS will allow the iscsi solution meet its latency August 08 Page 7

8 and throughput requirements. In general, QoS should only be used when different types of network traffic are sharing the same physical network with iscsi traffic. The PowerConnect 54xx Series of switches provides a special iscsi Optimization mode that was developed to provide, in addition to some other optimizations, iscsi traffic with priority over other LAN traffic types. Be aware that this special iscsi Optimization mode is solely for the support of the PowerVault MD3000i product and is NOT to be enabled when the PowerConnect 54xx Series is used in a EqualLogic PS SAN. Instructions on configuring Quality of Service settings can be found at Dell s support site by using the following link: Configuring iscsi Mode on PC54xx Series Switches Flow Control Flow Control allows the receiving host or array to instruct the sending host or array to slow the stream of data. This can happen when the data receiver senses that it is being overwhelmed by data packets. Each Ethernet port has some dedicated memory called a buffer that allows it to temporarily store incoming Ethernet data while the application on the host processes that data. When the buffer gets full or more accurately, reaches a specific threshold of packets queued up in the buffer then the receiving host needs to have the ability to ask the sender to pause before sending more data. This gives the receiving host the opportunity to process the data already in the buffer. The receiver asks the sender to quit sending additional data packets sending "pause frames" to the packet sender. This pause frame causes the sender to wait until the sender sends an acknowledgement before the next packet transmission. This allows the receiver to process its backlog so it can later resume accepting input. What would happen if Flow Control were not available? For Ethernet, packets would continue to be sent to the receiving port, but there would be no room for the packets to be temporarily stored. The receiving port would simply ignore these incoming packets. Now, this action of ignoring incoming Ethernet packets might be a cause for alarm, but Ethernet and the higher layer communications protocols, such as TCP and IP work together to have those lost packets retransmitted to the receiving host. The problem with allowing this to happen is not the possibility of lost data, but a problem of perceived poor performance of the network. It takes time to determine that packets have been dropped, request the retransmission of those missing packets, and then actually send those missing packets of data to the receiving host. In fact, it takes much more time to do this process, that to use Flow Control to ask for a temporary pause in the data packet transmissions from the sending array or host. Therefore, Flow Control is very important to a well designed and high-performance iscsi Ethernet infrastructure. Flow Control is configured on a port by port basis on the PowerConnect 54xx Series. Instructions on implementing Flow Control can be found at Dell s support site by using the following link: Configuring Individual Port Settings for the PC5400 Series Switches August 08 Page 8

9 Storm Management An Ethernet traffic storm occurs when an excessively large number of data packets get transmitted causing excessive network traffic. Many switches, including the PowerConnect 54xx series of switches, have traffic storm control features that prevent ports from being disrupted. There are three types of storms that can occur on a network: broadcast, multicast, or unicast. Storm management features typically work by discarding network packets when the traffic on an interface reaches a percentage of the overall load (usually 80 percent, by default). Because iscsi hosts and arrays can at times utilize 80+ percent of the available network bandwidth, it is very common that the switch management software will interpret the high utilization as a traffic storm. With Storm Management enabled, the switch will start dropping Ethernet data packets, thus causing a high number of retransmissions, and thus an increased amount of latency to the iscsi connection between the host and array. To prevent this from happening, Storm Management must be disabled for iscsi networks. In particular, iscsi traffic takes the form of unicast transmissions i.e. one to one communications between a host and an array and unicast storm control should be disabled on all switches in the iscsi network. In contrast, all other forms of storm control should be enabled. Configuring Storm Control Settings for the PC54oo Series Switches Port Forwarding Modes In a typical Non-stacking Ethernet switch, the data packets coming into the switch must be analyzed to determine the destination address and then forwarded to that destination Ethernet port. There are several standard methods for performing this process, collectively described as Port Forwarding Modes. There are four forwarding methods a switch can use. The first method is the most basic of the four and the second and third methods are performance-increasing methods. The fourth method is a dynamic switching method that will, as the situation requires, use one of the first three. Here are the descriptions of each method. 1. Store and Forward: The switch buffers the incoming packet, typically, checks the packet to ensure that the frame is complete, determines the destination address and then forwards it on to the destination port. 2. Cut Through: The switch reads only the part of the Ethernet packet that contains the hardware address before starting to forward it. There is no error checking with this method. 3. Fragment Free: Fragment free checks the first 64 bytes of the Ethernet frame, where addressing information is stored. Error detection is performed by higher layers of the networking protocol, such as at the IP protocol, rather than at the physical Ethernet layer. 4. Adaptive Switching: A method of automatically switching between the other three modes. So, what does this mean for an iscsi network? A non-stacking switch configured to use Store and Forward, must wait for the entire Ethernet packet to be received and stored in the port buffer before it can be retransmitted to the destination. This process contributes to the switch s overall latency, which can reduce the performance of the iscsi storage solution, but only for non-stacking switches that do not August 08 Page 9

10 have the bandwidth to support wire speed transfers. The PowerConnect 5400 series of switches perform all transfers at full wire speed, with forwarding speeds of 71.2 Gbits per second and a latency of 2.6 ms allowing the switch to be non-blocking with respect to Ethernet packet transmission. Port forwarding on the PC54xx Series Switches is fixed for the Store and Forward mode. Jumbo Frames The most important aspect of any SAN is the ability of that SAN infrastructure to store or retrieve as much information as possible in the shortest time possible. Latency incurred due to data storage activities will adversely affect your application s performance. For an iscsi based storage infrastructure, that utilizes standard Ethernet protocols, using Jumbo frames may be the difference between a storage solution with average data throughput and high latency and one with high performance and low latency. What are Jumbo Frames? To understand the concept, you need to know what a standard frame is. Basically, each Ethernet packet, or frame, will have a maximum size that can be understood by all Ethernet devices. The maximum size for a standard Ethernet packet is 1,518 bytes, of which 18 bytes are used by the Ethernet protocol for addressing, and error detection, and 1,500 bytes are actually available for data the payload of the Ethernet packet. Jumbo Frames basically have a larger payload. Instead of 1,500 bytes of data, the jumbo frame can transmit a maximum payload of 9,000 bytes of data, or 6x the size of a standard Ethernet frame. To take advantage of Jumbo Frames, all devices in the network path between servers and the PS Series group including the switches and the NICs used to access volumes must have Jumbo Frames enabled. Switches configured for Jumbo Frames will support both standard Ethernet frames and Jumbo Frames. However, if a NIC is configured for Jumbo Frames, but the switch is not, you may experience inconsistent behavior. The switch will function properly if the frames are small, but once the NIC attempts to send frames larger than 1500 bytes, the switch will not be able to handle the frames and will drop them. Also, if some switches are configured for Jumbo Frames, but others are not, you may experience inconsistent behavior if routing changes occur after the connection has been established. Should you use jumbo frames? The answer to this question is it depends. There are several factors that could affect the performance of an Ethernet network and determine whether the use of jumbo frames actually will provide better network performance, especially whether the switch supports advanced port forwarding methods and the threshold setting used by the switch vendor for Flow Control. If a switch only supports Store and Forward as the port forwarding method, as already discussed, the switch must wait for the entire packet to be received before it can forward the packet to its next destination. Since jumbo packets are 6 times as big as a standard packet, that means the switch must wait 6 times as normal before it can forward the packet to the next destination. Of course, each packet is sending 6 times the amount of data. If your storage requests are small in size as may be the case in many transactional database applications, jumbo frames may not improve performance and in fact, may degrade performance. August 08 Page 10

11 The switch vendor s implementation of Flow Control can also determine whether jumbo frames will enhance your iscsi solution s performance. As discussed in the flow control section earlier, each vendor hard-wires a threshold that, once reached, the switch can tell the sender to pause its transmissions. Switches with a low amount of buffer space per switch port will send these pause requests more often than switches with larger sized buffers or those that can dynamically allocate buffer memory to heavily used ports. Also, if the maximum size of the buffer space cannot wholly contain one or more jumbo frames, then the use of jumbo frames will be adversely affected by the use of jumbo frames on that switch. Instructions on enabling Jumbo Frames can be found at Dell s support site by using the following link: Configuring Jumbo Frames for the PC54xx Series Switches Port Aggregation Port Aggregation is a feature that allows the linking of a group of switch ports together to form a single, logical port called a Link Aggregated Group (LAG). Port Aggregation multiplies the bandwidth, increases port flexibility, and provides link redundancy between the two devices connected to each end of the LAG. LAGs are very important to the scalability of a SAN based on PS Series arrays. In a fully redundant array infrastructure, two switches are required and those switches must be connected to each other through Inter-Switch Links (ISLs) to ensure that every EqualLogic array has full bandwidth to each switch for each active port on each array. As discussed earlier in the section on switch sizing, as the number of arrays in the SAN increases, the number of ISLs increases. The PowerConnect 54xx Series switch supports up to eight LAGs per switch, each consisting of a maximum of eight ports, thus providing a maximum of 8 Gbits per LAG. Since each PS Series array in a SAN has 3 active Gbit ports, each array requires 3 Gbits of bandwidth between the two switches. To simplify management of the ISLs you want to create a LAG to logically create a single ISL pipe. For our two array example earlier, the pipe between the switches would need to be 6 Gbits in size, so a LAG would be created using 6 ports on each switch as illustrated in Figure 2. Other vendor s switches may support more or fewer links per aggregation group. Fewer links per group would mean that switch would not be able to support as many arrays as the PowerConnect 54xx, while more links per aggregation group would mean that switch would potentially support more arrays in a two switch configuration. 6x 1Gb LAG Figure 2 August 08 Page 11

12 PowerConnect =GREEN 10/100=ORANGE LNK/ACT FDX PWR CONSOLE DIAG LNK/ACT RPS FDX 1000=GREEN 10/100=ORANGE , N, 8, PowerConnect =GREEN 10/100=ORANGE LNK/ACT FDX PWR CONSOLE DIAG LNK/ACT RPS FDX 1000=GREEN 10/100=ORANGE , N, 8, PowerConnect =GREEN 10/100=ORANGE LNK/ACT FDX PWR CONSOLE DIAG LNK/ACT RPS FDX 1000=GREEN 10/100=ORANGE 9600, N, 8, 1 EqualLogic Storage and Non-stacking Switches Instructions on configuring a Link Aggregation Group can be found at Dell s support site by using the following link: Configuring Link Aggregation Groups for the PC54xx Series Switches Spanning Tree Protocol Spanning Tree Protocol (STP) is a standard process organizing a Non-stacking network infrastructure when that switched network is made up of 2 or more switches that are interconnected. Due to the mostly unstructured way that switches could be interconnected, any time there are more than two switches in a LAN infrastructure there is always a possibility that there is more than one path between a port on one switch and any second port on another switch. This creates a loop in the network, which in its simplest form is illustrated in Figure 3. If such a loop exists, a broadcast from a host could be duplicated and broadcast multiple times creating a network packet storm that would prevent the network from operating as designed. Figure 3 STP and its more modern variations, such as Rapid STP, were designed to prevent these loops from being created within the network infrastructure. STP accomplishes this by identifying only one of potentially many path as the active path for communications between switches and it blocks any ports that connect to any other path that is determined to be a duplicate path to the same destination switch. A detailed discussion of STP is beyond the scope of this paper, but further information can be found as part of the 802.1D-2004 standard at the IEEE website. Because of the limitations described earlier in this paper with respect to non-stacked Ethernet switched environments when used with EqualLogic PS Series arrays, it is not recommended that the non-stacked iscsi infrastructure go beyond two switches in size. While it is technically possible to go beyond 2 switches, the infrastructure will no longer be fully redundant and if a switch in the larger infrastructure were to fail, you cannot ensure that there will be a clear path between a specific host and an array node, or a clear path between any two arrays in the SAN. This could cause catastrophic failure SAN. For SAN solutions that will need to grow beyond the two-switch non-stacked switch SAN, it is recommended that the user implement more advanced stacking switches that are designed to support multiple switch environments and which can be expanded dynamically if your storage need exceed the size originally planned and implemented. August 08 Page 12

13 For a non-stacked two-switch infrastructure, STP does not provide any value and therefore should be disabled on the switches. If STP is implemented it should be the Rapid STP version of the Spanning Tree Protocol and should be used on any switch ports or LAGs that are being used as ISLs between switches. Any ports that will be attached to hosts or EqualLogic arrays, should have STP disabled. If STP is enabled on ports connected to hosts or arrays, then they should have the port fast setting enabled. Instructions on enabling or disabling Rapid Spanning Tree Protocol settings can be found at Dell s support site by using the following link: Configuring Spanning Tree Protocol for the PC54xx Series Switches Conclusions Understanding the requirements for a specific application are key to designing a high performance SAN solution. Each application will have specific requirements in a storage solution in terms of performance. This performance is defined in terms of throughput ( in MB/sec ) and in speed of response ( in IO/sec ). Knowing what the application requires will define storage criteria such as number of disk drives, size of disk drives, and amount of bandwidth required to get optimal performance out of the application. The Dell EqualLogic PS Series is a high performance, Enterprise class iscsi storage solution. The ability to incrementally expand your SAN, not only in terms of storage capacity and number of disk drives, but also in terms of performance, is a key differentiator between the PS Series and other storage solutions. To take full advantage of the performance scalability, the network infrastructure connecting the SAN to its hosts should be well designed and properly sized to meet your current and future needs in terms of storage growth and host connectivity growth. Non-stacking switches such as the PowerConnect 54xx Series provide Enterprise functionality in an inexpensive package. Providing key features that can help to optimize the network infrastructure to support iscsi, the PowerConnect 54xx Series is a great solution for the small EqualLogic SAN when properly sized and configured. SANs that have growth potential beyond the scope described in this paper should consider implementing more advanced stacking switches such as the Cisco Systems Catalyst 3750 family. August 08 Page 13