HP StorageWorks Enterprise File Services Clustered Gateway performance and scalability with HP StorageWorks XP12000 Disk Array white paper

Transcription

1 StorageWorks Enterprise File Services Clustered Gateway performance and scalability with StorageWorks XP Disk rray white paper Executive summary... Test environment... StorageWorks XP Disk rray... StorageWorks EFS Clustered Gateway... File services clients... Network... Test approach... Tools... Procedures... 6 Test results... 8 For more information...

2 Executive summary High file system performance is often a requirement for the purchase of a storage solution, and performance scalability is a key differentiator for the StorageWorks Enterprise File Services (EFS) Clustered Gateway product. While a large percentage of the market for this product uses middle tier storage, such as the StorageWorks Enterprise Virtual rrays (EV), enterprises requiring fast storage often purchase higher end disk arrays, such as those in the StorageWorks XP family. Those customers who want high-performance storage and file services will likely purchase an XP disk array and two or more nodes of the Clustered Gateway product. This white paper describes a performance testing environment, testing approach, tuning considerations, and test results used to demonstrate the performance scalability of file services using multiple Clustered Gateway nodes, an StorageWorks XP Disk rray, and the required network infrastructure. Test environment The following figure shows the network and storage area network (SN) environment used for this performance testing. Each Clustered Gateway node used three GigE interfaces: one for the private cluster network (used for heartbeat, monitors, and DLM traffic) and two for physically separate networks (PERF_ and PERF_). These networks were configured as VLNs at the Ethernet switch. Note that the NFS clients were heterogeneous, both in terms of hardware platform and operating system (OS). While this is not optimal for generating the best performance results, it is typical that a variety of systems would be accessing storage through the Clustered Gateway product.

3 Figure. XP Disk rray test environment EFS Clustered Gateway/XP Performance Testing Environment perf DL8 G SLES perf DL8 DL8 G SLES 9 CGW.., patch FC Fabric R EMOVLE SPR EMIRROR ED RID OR D G priv DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o Lia n t D L 8 DL8 R EMOVLE SPR EMIRROR ED RID OR D G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G h p P r o L ia n t D L 8 DL8 REMOVLE SPREMIRROREDRID ORD G DL8 G SLES T a pe ch c h hp Pr ol iant D L8 G D uplex Simplex U ID Ta pe ch ch hp Pr ol iant D L8 G D up lex Simp lex U ID Ta pe ch c h hp Pr ol iant D L8 G D uplex S t o r a g e W o r k s x p d i s k a r r a y h p S t o r a g e W o r k s x p d i s k a r r a h p y S t o r a g e W o r k s x p d i s k a r r a y h p S t o r a g e W o r k s x p d i s k a r r a Simplex U ID h p y RE D Y L RM MESS G E Ta pe ch c h hp Pr ol iant D L8 G U ID d i s k a r r a y D uplex x p o r k s S t o r a g e W h p Simplex U S H P Ta pe ch ch hp Pr ol iant D L8 G XP D uplex Simplex U ID DL SLES CGW Node Configuration: - Dual core.6 GHz Opteron procs. - 8 G Memory - GigE ports: internal, PCI-X - Gb/s FC ports (Qlogic) XP Configuration: - 6 CHIP P-P ports ( Gb/s) G LDEVs (D+D) - LDEVs presented to each host port - 8 paths for each LDEV per node Lp lades RHEL, Upd. (7 blades used) hp L6p hp L6p hp L6p hp L6p hp L6p hp L6p hp L6p

4 StorageWorks XP Disk rray The following figure shows the disk array and controller configuration used for the performance testing. For performance optimization, disk groups managed by a particular Disk Control frame (DKC) are spread across the clusters and striped vertically. The LDEVs created for these tests are all RID (D+D). Higher RID levels (for example, RID ) would have introduced unwanted parity calculation overhead. Each LDEV is.7 G and presented to each of the eight Clustered Gateway nodes, utilizing a total of 6 -Gb/s Fibre Channel P-P CHIP interfaces on the DKC. total of 6 LDEVs were created and presented across the 6 CHIP interfaces to the eight nodes. Figure. XP Disk rray/dkc configuration StorageWorks EFS Clustered Gateway Due to the need to show scalability, the various performance tests involved between one and eight nodes. The testing included the following node hardware/software specs: ProLiant DL8 G Server two dual-core.6-ghz Opteron processors 8-G RM x GigE ports: two internal, two used on a quad port PCI-X NIC x -Gb/s FC ports (QLogic) SLES 9, EFS Clustered Gateway.., patch Each Clustered Gateway node was presented with all 6 of the XP LUNs used in this testing, as required by the Clustered Gateway software. These presentations were specifically made to ensure that each server used two CHIP ports and no two servers used the same CHIP ports. Furthermore, the LUNs presented through each CHIP port was presented to different H ports on the server, creating a one-to-one relationship between the CHIP port and server H port, with LUNs presented per CHIP port/h port combination. These LUNs were imported into the cluster, and PolyServe Dynamic Volumes were created, striping across two LUNs each (with a 6-K stripe size). We verified that the two LUNs comprising a volume had different paths (presented on different CHIP ports) in all cases, thus ensuring that any activity to any Dynamic Volume would utilize both H ports on

5 the Clustered Gateway server performing the activity, and both CHIP ports accessed by that server. Upon volume creation, each volume had a PSFS file system created upon it, and each server mounted four of the PSFS file systems. The file systems were then added to a single Export Group, and associated with a Virtual NFS Service (one Virtual NFS Service was created for each physical interface on each server attached to the PERF_ and PERF_ networks). ll exports were created with minimal security considerations and for maximum performance (exported to the world, no_root_squash, nohide, no_wdelay, async, and so on). File services clients The heterogeneous NFS client environment for this testing included the following sets of servers: x ProLiant DL8 G Server (SLES ) x ProLiant DL8 G Server (SLES) x ProLiant DL Server (SLES ) 7x ProLiant Lp lades (RHEL m, Update ) These servers were used predominantly as NFS clients for the client-side testing. Each client node mounted four file systems exported from the Clustered Gateway cluster, with client pairs mounting the four file systems served from a single Clustered Gateway server across both the PERF_ and PERF_ networks, and automated iozone tests were run simultaneously and remotely from one of the clients. Network ecause of the high port count requirements for this test environment, two Cluster Platform ProCurve 6zl switches were used, with all four -Gb/s optical trunks used as an uplink. Two separate VLNs were used (PERF_ and PERF_), and these VLNs were tagged at the uplink. Test approach Tools The network testing was performed with netperf. This is a network test tool that performs a number of different tests; the only one used in this case was the TCP stream test. This is an approximation of the NFS traffic that would be generated by the clients. For netperf to operate correctly, a server must be running on the remote node to receive the traffic generated during the test. The test is unidirectional, and was performed from the client to the server, emulating NFS write traffic only. Command line: Clustered Gateway Server: netserver (no options) Client: netperf -fm -H${hostname} Options: -fm Specifies that out should be portrayed in megabytes, specifically power of two rather than power of notation (that is, divided by,8,76 as opposed to,,). -H${hostname} Specifies the target for the test. In this testing the clients were distributed evenly between the servers. For more details, see the Procedures section. URL:

6 The tool used for performing both local and NFS testing was iozone. This is a file system performance test tool, again with many modes of operation. For this testing, the iozone tool was only used to perform large block size streaming writes and reads through the use of the following command line, with the following options explained. Command line: Write: biozone -i -e -+n -s m -r m -t ${run_threads} -w -+m./machinefile Read: iozone -i -e -+n -s m -r m -t ${run_threads} -w -+m./machinefile Options: -i/-i Controls the type of test is a write test, is a read test. -e Specifies that the data flush (fsync/fflush) should be included in the timing calculations. This ensures that data being written from clients is not considered written until it is flushed from the client buffer cache. -+n Chooses not to run re-test operations. This avoids re-writing or re-reading the files after the pertinent write or read operation. This creates a more efficient use of test time. -s m Is the size of each individual file used in the test. Each thread will write or read a single file of this size. This is set to M for this testing. -r m Is the record size. Each individual write or read request generated by the test program will be of this size. This is set to M for this testing. -t ${run_threads} Is the number of threads that are used in the test run. This value was variable throughout the testing performed, and was set to eight threads per client used. -w Specifies that the file used in testing should not be deleted after the test is finished. This ensures that after a write test, the file created is available for reading. fter the read test, the files created were deleted manually. For more details, see the Procedures section. -+m./machinefile Specifies the file containing the remote execution details. This is used by iozone to distribute process across multiple remote systems. For more details, see the Procedures section. URL: Procedures The intention of this testing was to show the scalability of the Clustered Gateway solution when backed by an XP Disk rray. n initial set of tests were performed against a single server to determine the optimum client load for maximum single server throughput. When this was determined, the scalability testing was performed using the single server test case as a building block for each additional server tested. t each stage of the testing, the network was first verified by using the netperf tool, and after any discovered issues were fixed, the iozone tool was used to determine the throughput figures, as detailed in the Test results section. s previously stated, the initial phase of testing was to size the initial building block that would be the basis of the full set of scalability tests. To do this, a single server was tested with a variety of variables to determine optimum throughput with the available hardware. The key variables tested were: Number of NFS client nodes accessing the PSFS volumes on the server Number of threads running on each client node used in the test Number of PSFS volumes accessed on the server by each test 6

7 full matrix of these variables was constructed, and all of the tests were run through an automated script to ascertain the varying throughput figures for each combination in the matrix, as shown in the following table. Table. Performance test results (NFS clients) M/s Scaling factor Scaling coefficient # servers Write Read Write Read Write Read % % % % % % % % % % % 99% % 99% % 98% From the figures obtained, a client load of two clients running 6 threads accessing four PSFS volumes on each Clustered Gateway server was chosen. Now the scalability tests could begin. efore running the throughput tests, a network throughput and scalability test was run to determine that the network hardware was capable of full throughput to the Clustered Gateway servers. This testing showed deficiencies with the network due to poor network cables. Through analysis of the network switch port error counters, the bad network cables were discovered and replaced. further round of network testing verified these cables and allowed the throughput testing to begin. gain, an automated script was created to ensure that the exact tests performed could be replicated easily. Other automated scripts were also generated to handle the mounting of the file systems on the client nodes and test setup/tidy-up procedures, again with a view to ensuring that all of the results obtained could be easily repeatable. The key to testing successfully was to ensure that all of the clients had mounted file systems correctly and in an ordered way, and that each iozone thread on each client was accessing the appropriate file system. Furthermore, each test thread was to use a file in a unique directory, to reduce the effects of multi-thread access to the PSFS volumes. lso, to ensure that the clients did not access their local cache when reading the file, each client would read the files created by his or her partner client. fter the automated test script was written, and verified to produce the required behavior, it was executed to obtain the results detailed in the Test results section, as shown in the following figure. 7

8 Test results Figure. Clustered Gateway servers Clustered Gateway Single Server I/O Throughput via NFS Megabytes per second (M/s) Data verage of write verage of read # clients / # PSFS volumes under test per client / # threads per client nodes luns threads The preceding figures show the throughput figures from the initial testing performed for determining the optimum load to use for the full multi-client multi-server NFS testing. The figures clearly show that as the load increases by the Clustered Gateway server using more clients and more client-side I/O threads, the throughput increased. This would be an expected behavior, as would the rough plateau shown when the server is close to link speed. The data shows that the ideal test load would have been four clients accessing all four file systems on a single Clustered Gateway server, but due to hardware limitations within the test environment, two clients each running 6 threads was the maximum that could be used for the full-scale testing. However, there were a number of problems with the test tool, particularly with scaling. During the full scalability testing, it was discovered that at larger client counts using 6 threads per client would cause a failure in the test tool. Due to this finding, the full-scale testing was performed with eight threads per client, as opposed to 6. lso, attempts to scale the test tool used (iozone) above 6 threads per client, even when using just a single client, were unsuccessful, hence the cut-off point of 6 threads per client in the preceding results. 8

9 Table. Performance test results (NFS clients) M/s Scaling factor Scaling coefficient # servers Write Read Write Read Write Read % % % % % % % % % % % 99% % 99% % 98% Figure. Read/write scalability graph luns Clustered Gateway I/O Throughput (via NFS) Megabytes per second (M/s) Data 98 Cluster Size (Servers) verage of read verage of write servers The preceding figure and following table, show very linear performance scalability as nodes are added to the cluster. In addition, the high scaling coefficient for both reads and writes indicate that, although the scaling coefficient starts to drop off, there should be reasonable scalability, up to a 6- node cluster. This drop can be attributed to the heterogeneous client mix and lack of clients available for performing testing with. It is expected that with a larger number of clients, with care and consideration given to their OS and specification, that the scaling up to 6 Clustered Gateway servers would be linear with a single XP Disk rray. 9

10 More importantly, these numbers confirmed that with this performance configuration, we could comfortably achieve close to M/s, per node. This level of performance caused approximately 6% CPU utilization of the Clustered Gateway CPUs, suggesting that the bottleneck was due to the available throughput of the TCP network interfaces utilised for the testing. This is proven by the initial series of iozone tests (writes only) performed to the local file systems on the Clustered Gateway servers, as shown in the following figure. Table. Performance test results (local file system) # servers M/s Delta Scaling factor Scaling coefficient % % % % % % % % Figure. Local node write performance scalability type cumulative luns Clustered Gateway I/O Throughput (Local PSFS) Megabytes per second (M/s) Data 9.9 Servers verage of write Max of delta nodes

11 The dramatically better per-node performance in this local test underscores the fact that the network is the bottleneck, when working with NFS, as would be expected. These results show a total throughput of approximately.6 Gb/s to the XP Disk rray. The specification for the array states that the large sequential IO throughput of the array is greater than 9 Gb/s. This would suggest that even a fully configured Clustered Gateway system of 6 nodes would not saturate the XP Disk rray with the configuration used for this testing. The poor scaling shown is due to individual test run fluctuations of up to % on each server. These effects can be negated in future testing by ensuring to run for a longer period of time by using larger test files, and repeating the tests multiple times and taking the average over a minimum of five runs. Similarly, it is expected that this would level the delta between successive runs to a nearly constant value. Finally, it should be noted that none of these tests saturated the XP Disk rray. Further test design is recommended to isolate the point where the array is saturated and performance scalability flattens. This kind of testing is likely to require more Clustered Gateway nodes, and should be performed locally on the nodes, taking client network inefficiency out of the equation.

12 For more information 7 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. shall not be liable for technical or editorial errors or omissions contained herein. -7ENW, ugust 7