Networking best practices with IBM Storwize V7000 and iscsi Reference guide for network and storage administrators

Transcription

1 Networking best practices with IBM Storwize V7000 and iscsi Reference guide for network and storage administrators Shashank Shingornikar IBM Systems and Technology Group ISV Enablement September 2013 Copyright IBM Corporation, 2013

2 Table of contents Abstract... 1 Intended audience... 1 Scope... 1 Introduction to the IBM Storwize V7000 system... 2 IBM Storwize V7000 configuration... 3 IBM Storwize V7000 terminology... 3 iscsi overview... 4 Understanding iscsi basics... 4 Planning network... 5 Ethernet network design... 5 Ethernet network topologies... 8 Configuring iscsi on Storwize V Creating and mapping Storwize V7000 volumes using iscsi Configuring ESX Lab setup Storwize V ESX host details Lab topology Same subnet configuration Multiple subnet configuration Conclusion Summary Acknowledgements Resources About the author Trademarks and special notices... 21

3 Abstract This paper explores the use of the Internet Small Computer System Interface (iscsi) protocol provided by the IBM Storwize V7000 storage product as an essential component of the infrastructure solution. The paper highlights the network considerations and best practices that must be made when exposing storage logical unit numbers (LUNs) to an application server. Although VMware ESX is the platform used for writing this paper, the core networking part is independent of the type of platform or operating system used. Intended audience This technical report is intended for: Scope Customers and prospects looking to implement iscsi protocol of IBM Storwize V7000 system in their existing infrastructure. Network or storage administrators seeking detailed information on networking best practices when working with iscsi protocol of IBM Storwize V7000 system. Although IBM Storwize V7000 is perfectly capable of supporting both Fibre Channel (FC) and iscsi protocols, in this paper, the discussion is limited to storage exposed over iscsi only. This technical report provides: Detailed instructions for the implementation of iscsi-based network configuration with IBM Storwize V7000 storage. Configuration aspects from ESXi perspective when using iscsi Networking best practices such as use of virtual local area network (VLAN) This technical report does not: Discuss any performance impact and analysis from a user perspective. Replace any official manuals and documents from IBM on Storwize V7000. Replace any official manual or support document provided by VMware. 1

4 Introduction to the IBM Storwize V7000 system The IBM Storwize V7000 system combines hardware and software to control the mapping of storage into volumes in a SAN environment. The Storwize V7000 system provides many benefits to storage administrators, including simplified storage administration, integrated management of IBM servers and storage, and enterprise-class performance, function, and reliability. Figure 1: IBM Storwize V7000 The Storwize V7000 system includes rack-mounted units called enclosures. Each enclosure can be a 12- or 24-drive model, and has two canisters and two power supplies located at the back. There are two types of enclosures: control and expansion. A system can support more than one control enclosure, and a single control enclosure can have several expansion units attached to it. Node canisters are always installed in pairs as part of a control enclosure. Each control enclosure represents an I/O group. Any expansion enclosure that is attached to a specific control enclosure also belongs to the same I/O group. The system also includes an easy-to-use management graphical user interface (GUI), which helps you to configure, troubleshoot, and manage the system. For more information about the IBM Storwize V7000 system, refer to the URL: pic.dhe.ibm.com/infocenter/storwize/ic/index.jsp 2

5 IBM Storwize V7000 configuration This section explains the configuration of the various components that were involved in testing. It includes: SAN configuration, managed disks (MDisks) configuration, storage pools, volumes, and hosts. This section explains the common terminology used in Storwize V7000. IBM Storwize V7000 terminology Table 1 lists the IBM Storwize V7000 terminology used in the paper. For an entire list of the Storwize V7000 terminology, refer to IBM Storwize V7000 Introduction and Implementation Guide at ibm.com/redbooks/redpieces/abstracts/sg html?open IBM Storwize V7000 term Control enclosure Expansion enclosure Node canister Host mapping Internal storage MDisk Storage pool Thin provisioning or thin provisioned Volume Extent Definition A hardware unit that includes the chassis, node canisters, drives, and energy sources including batteries. A hardware unit that includes expansion canisters, drives, and energy sources that do not include batteries. A hardware unit that includes the node hardware, fabric and service interfaces, and SAS expansion ports. A controlling process in which hosts have access to only specific volumes within a cluster. Array-managed disks and drives that are held in enclosures and nodes that are part of the cluster. A component of a storage pool that is managed by a cluster. An MDisk is either a part of a Redundant Array of Independent Disks (RAID) array of internal storage or a SCSI logical unit (LU) for external storage. An MDisk is not visible to a host system on the storage area network (SAN). A collection of storage capacity that provides the capacity requirements for a volume. The ability to define a storage unit (full system, storage pool, or volume) with a logical capacity size that is larger than the physical capacity assigned to that storage unit. A discrete unit of storage on disk, tape, or other data recording medium that supports some form of identifier and parameter list, such as a volume label or I/O control. Each MDisk is divided into segments of equal size called extents. When a volume is created from a storage pool, the volume is allocated based on the number of extents required to meet the capacity requirements for the volume. Table 1: IBM Storwize V7000system terminology 3

6 iscsi overview This section provides and overview and explains the basics of iscsi. Understanding iscsi basics iscsi is an Internet Protocol (IP)-based storage networking protocol. The basic concept is to take SCSI commands and encapsulate them in Transmission Control Protocol/Internet Protocol (TCP/IP) packets to transmit data from the storage drives to the server. Because TCP packets can be lost and retransmitted, they do not have to arrive in order. iscsi also has to keep track of incoming packets to make sure that all of the SCSI commands are queued in the correct order. iscsi was originally started by IBM and was developed as a proof of concept in In March 2000, the first draft of the iscsi standard was presented to the Internet Engineering Task Force (IETF). IBM offered iscsi-based storage devices in July 2001 even before the iscsi specification was passed by the IP Storage Working Group in August The two fundamental concepts to explain in iscsi are initiator and target. The initiator is the server, or client, which initiates the request for storage block. The target is the storage device that accepts the request and responds by providing the storage blocks. Each initiator and target is given a unique iscsi name such as an iscsi qualified name (IQN) or an Extended Unique Identifier (EUI). iscsi is fundamentally a storage area network (SAN) protocol similar to Fibre Channel. The key difference is that FC uses a specialized (FC) network and iscsi uses TCP networks. This allows you to use a single network for storage data and other communication, as shown in Figure 2. Figure 2: iscsi overview 4

7 Planning network Plan in advance the network infrastructure and the storage network infrastructure that your deployment will require. This section describes various network configurations and the components involved. Maximizing iscsi SAN performance is a complex endeavor. High throughput is expected, as is a short response time for each I/O. The 1-Gigabit Ethernet typically tops out at a little more than 117 MBps. Some of the bandwidth is consumed by protocol and the maximum iscsi data throughput measures at about 110 MBps. Response time is a function of packet loss and network latency. Packet loss necessitates recovery by the TCP protocol. The recovery process is not fast, and by design, can sacrifice performance on the connection in recovery in deference to the health of the network in general. Network latency can be introduced by traversing wide area network (WAN) environments or through network components or end nodes that are overwhelmed with traffic. Network latency is an issue on its own, in addition to complicating recovery from packet loss. Poor network design and network components can cause packet loss and increase network latency. Also, a system, poorly configured for network conditions might not be able to maintain levels of throughput or response time requirements. Ethernet network design This section discusses some options available to the network designer to deal with the problems of packet loss and latency that occur in a congested and oversubscribed network. These are only guidelines to be considered while designing the Ethernet network. Network infrastructure bandwidth Network designs must take into account the bandwidth requirements of the applications that run over the network. For networks that include iscsi storage, the network infrastructure must take into account the bandwidth requirements of all iscsi initiators and targets. Switches should be evaluated for required proper backplane bandwidth. Buffering within network Although sufficient bandwidth is available, many targets responding back to one initiator could result in egress port congestion problem. A typical example of such a situation might be multiple 10 Gb links at the target and single 1 or 10 Gb link at the initiator. As a result, when the initiator issues large read requests, they are easily served by the targets, delivering larger data set to the initiator. But the single initiator port in this case is not able to accept all the data and thus results in dropping the packets. 5

8 Ethernet switches implement buffering schemes or methodology; it may help knowing these before designing the iscsi storage network. Buffers in the switch can help the network tolerate short bursts of traffic that might temporarily oversubscribe a link and drop packets. TCP maximum segment size tuning Larger value for maximum segment size (MSS) can improve TCP throughput. If the networking equipment supports jumbo frames, configuring a large Ethernet maximum transmission unit (MTU) can improve throughput and improve recovery of lost data. This is because recovering one dropped frame is faster than recovering from a series of dropped frames. A network switch can more readily drop jumbo frames than 1500-byte frames resulting in more recovery events. Also, if the switch is a store-and-forward device, enabling jumbo frames can increase network latency. TCP window size and window scale option For more efficient use of high bandwidth networks, a larger TCP window size might be used. The TCP window size field controls the flow of data and its value is limited between 2 and 65,535 bytes. The TCP Receive Window size of a connection endpoint is advertised in each TCP packet sent on the connection. Because the size field cannot be expanded, a scaling factor is used. TCP Window scale is used to increase the amount of allowed unacknowledged data outstanding on a connection when latency is introduced into a network. Allowing more data in the pipe utilizes the available bandwidth more effectively. Larger TCP window values increase the amount of data outstanding, which means that it also increases the amount of data that can be dropped. For this reason, latency and packet loss should be taken into consideration when determining the TCP window values to use. TCP timestamps The TCP retransmission algorithms work best when the round-trip time is accurate. The TCP Timestamp option enables the TCP protocol to calculate round-trip times more accurately and more frequently. A more accurate round-trip time can help the TCP engine recover from packet loss more efficiently. The TCP Timestamp option is negotiated between both ends of the TCP connection during connection establishment. Both sides must agree to use the TCP Timestamp option or neither side can use it. Environments that do not enable TCP Timestamp should not expect optimal recovery from packet loss situations. ESX / ESXi parameters The following sections briefly describe the ESX/ESXi parameters that can have an impact on the network. Flow control (network interface) To manage the pacing of data transmission on a network, the Ethernet flow control pause frames can be used. Sometimes, a sending node (ESX/ESXi host, switch, and so on) might transmit data faster than 6

9 another node can accept it. In this case, the overwhelmed network node can send pause frames back to the sender, pausing the transmission of traffic for a brief period of time. By default, flow control is enabled on all network interfaces in VMware ESX and ESXi. This is the preferred configuration. VMware knowledge base article ID: Queue depth (applies to both ESX/ESXi and Storwize V7000) The queue depth is the number of I/O operations that can run in parallel on a device. If you are designing a configuration for a large SAN, you must estimate the queue depth for each node in order to avoid application failures. If a Storwize V7000 node reaches the maximum number of queued commands, many operating systems cannot recover if the situation persists for more than 15 seconds. This can result in one or more servers presenting errors to applications and application failures on the servers. For more information, refer to following link from Storwize V7000 informationcenter: pic.dhe.ibm.com/infocenter/storwize/ic/index.jsp?topic=%2fcom.ibm.storwize.v7000.doc%2fsvc_configre cforlrgsans_1dcuwe.html VMware knowledge base article ID: Maximum transmission unit (MTU applies to both Storwize V7000 and ESX/ESXi) The Storwize V7000 system supports jumbo frame MTU packets. Ethernet jumbo frames improve performance in some environments by enabling 9000-byte Ethernet frames as opposed to the normal 1500-byte MTU frames. You can enable Ethernet jumbo frames by specifying an MTU of 9000 using the cfgportip commandline interface (CLI) command. When you change the interfaces to jumbo frames, you must also make the same changes to the switch ports that are connected to the Storwize V7000 ports. In addition, ensure that the path MTU from any client, whether management or data, that is communicating to the Storwize V7000 system is set to the identical maximum MTU size at which the Storwize V7000 system interfaces are configured. This prevents network packet fragmentation, which can greatly impact network throughput. Setting MTU on Storwize V7000 The usage of the cfgportip command is shown in the following example. Values for parameters such as iogrpid and id can be obtained by running the lsportip command. Syntax: cfgportip mtu <value> -iogrp <iogroupid> <id> Example: cfgportip -mtu iogrp 0 1 7

10 Setting MTU on ESX/ESXi To enable jumbo frames for software and dependent hardware iscsi adapters in the vsphere Web Client, change the default value of the MTU parameter: 1. Browse to the host in the vsphere Web Client navigator. 2. Click the Manage tab, and click Networking. 3. Click Virtual Switches, and select the vsphere switch that you want to modify from the list. 4. Click Edit Settings. 5. On the Properties page, change the MTU parameter. For more information, refer to VMware kb article: Customizing the round-robin path policy (ESX/ESXi) The round-robin path selection policy helps to balance the I/O across all the active paths. The path selection policy determines when to choose the next path to send the I/O, and this is based on certain criteria. By default, the path selection policy sends 1,000 I/O requests down the first path, then 1,000 I/O requests down the next and so on in a round-robin fashion. By lowering the default settings provided, good throughput performance can be achieved. This setting greatly depends on the load in a specific environment. Also, extreme care must be taken when modifying this parameter. For more information about the modification of input/output operations per second (IOPS), refer to: pubs.vmware.com/vsphere- 50/index.jsp#com.vmware.vcli.examples.doc_50/cli_advanced_storage.8.2.html?path=1_1_0_5_0_4# Ethernet network topologies This section discusses some of the topology options. In order to take advantage of the redundancy provided by Storwize V7000, it is expected that redundant components such as multiple network interface cards (NICs) and network switches exist in the topology design. 8

11 Single / Same subnets The following figure explains the single / same subnet design. Figure 3: Same / Single subnet design Figure 3 shows a two-node Storwize V7000 clustered system that is connected to a single subnet. Each node has two Ethernet ports, each of which is used for iscsi data transfers. One node in the system also acts as the system configuration node. In this example, port 1 on the configuration node provides the system management IP interface. The single subnet configuration is more vulnerable to traffic storms. A traffic storm occurs when packets flood the local area network (LAN), creating excessive traffic and degrading network performance. The traffic storm control feature (also called traffic suppression) from switches can be used to prevent disruptions on Layer 2 ports by a broadcast, multicast, or unknown unicast traffic storm on physical interfaces. It monitors incoming traffic levels over a traffic storm control interval and during the interval, compares the traffic level with the traffic storm control level configured by user. Alternatively, VLAN configuration can help separating physical ports, thus reducing the broadcast traffic. This is also suggested as part of best practices as this not only reduces the broadcast traffic but also helps isolate traffic between initiator hosts and the target Storwize V7000 system. 9

12 Multiple subnets and multipathing The following example explains multiple subnet design along with the multipath configuration. Figure 4: Multiple subnets Figure 4 shows a two-node Storwize V7000 system that is connected to multiple subnets. Each node has two Ethernet ports (port 1 and port 2) that are connected to different IP subnets. In addition, one node in the system also acts as the system configuration node, which provides alternate IP interfaces, again on different subnets for the system management interface. The definitions of the following key terms such as iscsi session and multipathing helps to better understand this configuration. iscsi session The group of TCP connections that link an initiator with a target form a session. TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one and the same target. 10

13 Multipathing To maintain a constant connection between a host and its storage, a technique called multipathing is used. This technique allows the use of more than one physical path that transfers data between the host and an external storage device. In this example configuration, host 1 does not use multipathing. A volume in the Storwize V7000 I/O group appears as four separate devices to host 1. The host (host 1) selects one device to perform I/O operations to the volume, which corresponds to a particular IP address at a Storwize V7000 node port. If a connection between the host and this Storwize V7000 port is broken, an I/O error is recorded on host 1 for that volume if I/O is in progress. No Storwize V7000 state changes or IP failover occur. Host 2 uses multipathing. A volume in the Storwize V7000 I/O group appears as a single device to the applications on host 2, even though the multipathing driver can detect four separate devices for each volume. The multipathing driver selects one or more of these devices during I/O. If the connection between the host and one Storwize V7000 node port is lost, the multipathing driver can select an alternative path to the Storwize V7000 I/O group. The I/O between the host and Storwize V7000 continue without errors. Host 2, however, has only one NIC and can therefore report I/O if the connection between that NIC and the network is lost. Host 3 uses multipathing and redundant NICs. This means that if an NIC fails, the multipathing driver can still find paths from the host to a volume in the Storwize V7000 I/O group and the application I/O can continue without errors. Because the NICs are connected to different IP networks, the overall configuration can tolerate a single network failing without I/O errors occurring on host 3. Host 3 configuration provides the best redundancy for the application and thus should be used as a best practice. Configuring iscsi on Storwize V7000 This section describes the iscsi configuration setup on Storwize V7000 systems created using GUI. You can start the iscsi configuration wizard by clicking Settings Network. 10 Gbps ports are configured for iscsi with a unique IP address per port on each node, thus providing redundant channels of communication per node (refer to Figure 5). The iscsi name assigned to the system is also displayed on this page and it cannot be modified, although an iscsi alias can be given in a free form text available. 11

14 Figure 5: iscsi configuration Both 1 GB and 10 GB Ethernet ports can be used for iscsi traffic, but only the 1 GB Ethernet ports can be used for management traffic All IP addresses (service and configuration) associated with a clustered-system Ethernet port must be on the same subnet. However, IP addresses associated with a node Ethernet port used for iscsi traffic can be configured to belong to different subnets. For more information about iscsi configuration, refer to: pic.dhe.ibm.com/infocenter/storwize/ic/topic/com.ibm.storwize.v7000.doc/svc_rulesiscsi_334gow.html Creating and mapping Storwize V7000 volumes using iscsi A volume is a logical disk that the clustered system presents to a host connected over a Fibre Channel or Ethernet network. Using volumes allows administrators to more efficiently manage storage resources. A storage pool is a collection of managed disks that jointly contain all the data for a specified set of volumes. First, the system divides each managed disk in the pool into extents storage blocks that are typically of equal size. Next, the administrator creates the volumes, a process that consumes the extents, and then maps them to the host objects. As a result, the clustered system presents the volumes to the hosts. Volumes are presented to hosts by host mapping, the process of controlling which hosts have access to specific volumes in the clustered system. Administrators can create logical hosts using Fibre Channel, Fibre Channel over Ethernet (FCoE), or iscsi technology. Volumes can then be mapped to a host. You have the ability to create different types of volumes, including mirrored, thin provisioned, and compressed volumes. Although it is possible to map the volumes using command-line interface (CLI), same can also be done through a GUI wizard, as shown in Figure 6. 12

15 Figure 6: New volume mapping (step 1) Selecting a new volume presents the following dialog box, allowing users to create a volume based on presets. Figure 7: Volume creation based on presets As shown in Figure 7, a new volume can be created based on predefined templates and mapped to the host. The advanced properties allow more flexibility on selection of a specific node for performing the I/O operation. 13

16 Completing the final step (as shown in Figure 8), the volume can be mapped to the host. Figure 8: Volume mapping Volumes are mapped to a host using the same host mapping mechanism as Fibre Channel attachment. A volume can be mapped to a Fibre Channel host or an iscsi host. Mapping a volume through both iscsi and Fibre Channel to the same host is not supported. Configuring ESX VMware ESX can support both hardware and software iscsi initiator. This paper discusses only about the software iscsi initiator. VMware uses a virtual Switch (vswitch) to work with the Storwize V7000 system. The vswitch contains the VMkernel (the kernel used by VMware to access the iscsi I/O) and the service console (the console through which management activities are performed), and uses a network port group. For the Storwize V7000 system to work with VMware, the IPs configured on the VMkernel must ping to and from the nodes in the clustered system to enable communication between VMware and the system. A vswitch is a software program that enables one virtual machine to communicate with another. Similar to a physical Ethernet switch, a vswitch directs communication on the network by inspecting packets before passing them on. Refer to the VMware documentation center for more details about configuring vswitches: pubs.vmware.com/vsphere- 50/index.jsp?topic=%2Fcom.vmware.vcli.examples.doc_50%2Fcli_manage_networks.11.4.html The software iscsi initiator on ESX can be configured from vcenter client GUI interface under the Storage Adapters section by selecting the specific adapter and clicking Properties. If the iscsi software adapter is not already present in the list, you need to add it first. Refer to following information center link for more details about how to configure VMware ESX/ESXi for iscsi host attachment. pic.dhe.ibm.com/infocenter/storwize/ic/topic/com.ibm.storwize.v7000.doc/svc_iscsi_vmware_cvr.html 14

17 Lab setup The lab environment under which this setup was tested had following components. Storwize V7000 The following table lists the Storwize V7000 configuration used in the exercise. Storage name Ifs3 Node model Storwize V7000 Micro code level 6.4 and Table 2: Storwize V7000 system details ESX host details The following table lists the ESX hosts used in the exercise. Server type IBM xseries 3650 M4 servers Processor Intel Xeon processor Memory 200 GB Operating system VMware ESXi Table 3: ESXi hosts details Lab topology The following figure shows the lab setup for iscsi configuration. Figure 9: Lab setup 15

18 VMware ESX 5.1 was used as iscsi initiator in the lab setup. Two 10 Gb Ethernet ports were configured for network traffic between the initiator and the target. A software iscsi initiator was used. For the data store, both policies round-robin and preferred path were tested. The target Storwize V7000 system consisted of two node canisters and multiple disk expansions with a variety of disks ranging from serial attached advanced SCSI (SAS), nearline serial-attached SCSI (NLSAS) and solid-state drives (SSDs). Two 10 Gb ports on each node canister were configured for iscsi IP. For this iscsi setup, the test team used a single virtual disk (vdisk) created from five MDisks. Each MDisk was created from nine individual drives of homogenous type. On the networking switches side, Blade Network Technologies RackSwitch G8264 and CISCO Nexus 5020 were used without any inter-switch link (ISL). Although, the team did not find any issues during testing, the choice of the switch is entirely dependent on the company policy. The team considered the following test cases. Some of these test cases do not consider redundancy; however, the team suggests that in order to maintain good availability of the production environment, a fully redundant environment should be envisioned. Same subnet configuration The following figure explains the test setup. Figure 10: Same subnet configuration Although, this is a valid configuration, it poses the following challenges. As only one switch is present in the configuration, it becomes a single point of failure (SPOF). In the event of switch failure, although, both ESX server and Storwize V7000 are available, switch failure can bring down the environment. 16

19 From ESX 8, storage paths are now visible as a single NIC on ESX can see all the four storage paths. Consider the case when a read-intensive operation is made from ESX, all the storage paths will send the data back. Overwhelmed with the data from storage, the switch now becomes the bottleneck and might start dropping the packets. In such a configuration, maximum amount of Ethernet broadcast is observed, which might lead to latency issues. The only way to reduce the Ethernet broadcast is by implementing VLAN configuration or subnetting the traffic. In the lab, the test team managed to reduce the Ethernet broadcast by implementing VLAN as explained in the following figure. Figure 11: Single switch and VLAN configuration Multiple subnet configuration The following figure explains the test setup. Figure 12: Multiple subnets 17

20 The multiple subnet network setups is more commonly seen in today s environment. The setup depicted in Figure 12 uses two separate network switches that are not connected to each other. A further isolation is made on the basis of VLANs in each switch, thus creating a controlled environment. Conclusion From the testing made in the environment, the test team observed the following points. Network traffic separation, either VLAN-based or subnet-based, can help reduce the broadcast traffic and lower packet drop. Adjusting the default queue depth setting along with IOPS settings for ESX round-robin policy can help improve the performance. The test team did not seen any significant changes in the performance when enabling the TCP header digest or TCP data digest or both. An end-to-end view of network helps understand and troubleshoot the iscsi issues. 18

21 Summary This paper described the considerations for networking practices when implementing iscsi-based storage from Storwize V7000. Acknowledgements Special thanks to Sanjay Sudam for helping out with setup configuration and providing inputs for test cases. Author also wishes to thank Shrikant Karve for his support and knowledge sharing on networking and iscsi. This paper could not have been completed without the valuable suggestions and validation efforts from the Storwize V7000 team members from the infrastructure team for making the hardware available. 19

22 Resources The following websites provide useful references to supplement the information contained in this paper: IBM Systems on PartnerWorld ibm.com/partnerworld/systems IBM Storwize V7000 Information Center IBM Redbooks ibm.com/redbooks IBM Power Systems Information Center iscsi storage research Cut-Through and Store-and-Foward Ethernet Switching for Low-Latency Environments Configuring traffic storm control on CISCO 7600 series routers iscsi and Jumbo Frames configuration on ESX/ESXi ( ) eid=&externalid= About the author Shashank Shingornikar is an IT specialist in the IBM Systems and Technology Group ISV Enablement Organization. He has more than 12 years of experience working with DBMS technologies, involved in high availability areas. You can reach Shashank at 20

23 Trademarks and special notices Copyright IBM Corporation References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. 21

24 Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 22