IM System Storage N series Universal SS and P abling Guide G27-3919-04
ontents Preface................................ v Supported features............................. v Websites................................ v Getting information, help, and service...................... vi efore you call.............................. vi Using the documentation.......................... vi Hardware service and support......................... vi Firmware updates............................. vii How to send your comments......................... vii SS cabling for new storage systems...................... 1 SS cabling rules............................. 1 General SS cabling rules......................... 1 Onboard SS port rules.......................... 4 Shelf-to- connection rules........................ 5 ontroller-to--connection rules...................... 6 ontroller-to-controller rule (N3240 systems only)................. 11 abling SS connections.......................... 11 ompleting the SS cabling worksheet..................... 12 SS cabling worksheet rules and template.................... 13 Sample SS cabling worksheet for a N6200 series................. 14 Sample SS cabling worksheet for an H pair (non N6200 series, N3240, or N3400 )..... 16 abling SS ports............................ 18 SS cabling examples........................... 24 Examples of SS cabling for H pairs with quad-port Hs............. 24 Examples of SS cabling for single-controller configurations with quad-port Hs...... 30 Examples of SS cabling for N6200 series systems................. 33 Examples of SS cabling for PM systems................... 36 Examples of SS cabling for N3600 systems................... 40 Examples of SS cabling for N3400 systems................... 45 P cabling for new storage systems..................... 49 P cabling rules............................. 49 General P cabling rules......................... 49 P connectivity rules.......................... 49 abling P connections.......................... 50 abling P on systems with external SS storage................ 50 abling P on PM systems with no external SS storage............. 56 P cabling examples........................... 57 Examples of P cabling for H pairs..................... 57 Examples of P cabling for single-controller configurations............. 61 P cabling examples for all single-controller configurations except N3400 systems..... 61 P cabling examples for single-controller N3400 systems............. 63 opyright IM orp. 2014 iii
Glossary............................... 67 ocument update record.......................... 71 opyright and trademark information..................... 73 Trademark information........................... 74 Notices................................ 77 iv IM System Storage N series: Universal SS and P abling Guide
Preface Supported features IM System Storage N series storage systems are driven by Netpp ata ONTP software. Some features described in the product software documentation are neither offered nor supported by IM. Please contact your local IM representative or reseller for further details. Information about supported features can also be found on the N series support website (accessed and navigated as described in Websites). Websites IM maintains pages on the World Wide Web where you can get the latest technical information and download device drivers and updates. The following web pages provide N series information: v listing of currently available N series products and features can be found at the following web page: www.ibm.com/storage/nas/ v The IM System Storage N series support website requires users to register in order to obtain access to N series support content on the web. To understand how the N series support web content is organized and navigated, and to access the N series support website, refer to the following publicly accessible web page: www.ibm.com/storage/support/nseries/ This web page also provides links to utosupport information as well as other important N series product resources. v IM System Storage N series products attach to a variety of servers and operating systems. To determine the latest supported attachments, go to the IM N series interoperability matrix at the following web page: www.ibm.com/systems/storage/network/interophome.html v For the latest N series hardware product documentation, including planning, installation and setup, and hardware monitoring, service and diagnostics, see the IM N series Information enter at the following web page: publib.boulder.ibm.com/infocenter/nasinfo/nseries/index.jsp opyright IM orp. 2014 v
Getting information, help, and service If you need help, service, or technical assistance or just want more information about IM products, you will find a wide variety of sources available from IM to assist you. This section contains information about where to go for additional information about IM and IM products, what to do if you experience a problem with your IM N series product, and whom to call for service, if it is necessary. efore you call efore you call, make sure you have taken these steps to try to solve the problem yourself: v heck all cables to make sure they are connected. v heck the power switches to make sure the system is turned on. v Use the troubleshooting information in your system documentation and use the diagnostic tools that come with your system. v Refer to the N series support website (accessed and navigated as described in Websites) for information on known problems and limitations. Using the documentation The latest versions of N series software documentation, including ata ONTP and other software products, are available on the N series support website (accessed and navigated as described in Websites). urrent N series hardware product documentation is shipped with your hardware product in printed documents or as PF files on a documentation. For the latest N series hardware product documentation PFs, go to the N series support website. Hardware documentation, including planning, installation and setup, and hardware monitoring, service, and diagnostics, is also provided in an IM N series Information enter at the following web page: publib.boulder.ibm.com/infocenter/nasinfo/nseries/index.jsp Hardware service and support You can receive hardware service through IM Integrated Technology Services. Visit the following web page for support telephone numbers: www.ibm.com/planetwide/ vi IM System Storage N series: Universal SS and P abling Guide
Firmware updates IM N series product firmware is embedded in ata ONTP. s with all devices, ensure that you run the latest level of firmware. ny firmware updates are posted to the N series support website (accessed and navigated as described in Websites). Note: If you do not see new firmware updates on the N series support website, you are running the latest level of firmware. Verify that the latest level of firmware is installed on your machine before contacting IM for technical support. How to send your comments Your feedback helps us to provide the most accurate and high-quality information. If you have comments or suggestions for improving this document, please send them by email to starpubs@us.ibm.com. e sure to include the following: v Exact publication title v Publication form number (for example, G26-1234-02) v Page, table, or illustration numbers v detailed description of any information that should be changed Preface vii
viii IM System Storage N series: Universal SS and P abling Guide
SS cabling for new storage systems For new storage system installations, you must understand the cabling rules before cabling the SS connections for the disk shelves. Note: lthough the cabling rules in this document apply to stretch Metroluster systems with SS disk shelves directly attached to the controllers, specific cabling procedures for stretch Metroluster systems using SS optical cables can be found in the onfiguring a stretch Metroluster system with SS disk shelves and SS optical cables document. Note: abling information for hot-adding or replacing SS disk shelves is in the Installation and Service Guide for your disk model. ll documents can be found on the N series support website (accessed and navigated as described in Websites) at www.ibm.com/storage/support/ nseries/. SS cabling rules You can cable SS disk shelves to storage systems that have SS Hs and/or onboard SS ports by applying the SS cabling rules. General SS cabling rules You need to know the general SS cabling rules governing supported configurations, cables, and SS port connections on the disk shelves. v ll H pair configurations must use multipath H, with the following exceptions: N3400 systems use single-path H. N3240 systems with no external SS storage can use single-path H, which requires no external cabling. However, the recommended configuration is to enable multipath H for the internal disks by connecting the SS ports of one controller module to the SS ports of the partner. Note: multipath H configuration having external shelves requires two ports on each controller for each stack of disk shelves. Even if the stack consists of one disk, two ports on each controller are required. v ll single-controller configurations must use dual-path, with the following exceptions: N3400 systems use single-path. opyright IM orp. 2014 1
N3240 systems have the internal disks cabled as single-path, but the system can use either single-path or dual-path connectivity to the external shelves; both are supported. ual-path provides greater resiliency, but when dual-path is used, ata ONTP 8.1.x issues occasional warnings to the console because mixed paths are detected. To avoid the warnings, single-path connections to the external SS storage are a supported option in these configurations. v The SS cables can be SS copper, SS optical, or a mix depending on whether or not your system meets the requirements for using the type of cable. If you are using a mix of SS copper cables and SS optical cables, the following rules apply: Shelf-to- connections in a stack must be all SS copper cables or all SS optical cables. If the -to- connections are SS optical cables, the -to-controller connections to that stack must also be SS optical cables. If the -to- connections are SS copper cables, the -to-controller connections to that stack can be SS optical cables or SS copper cables. v The cable QSFP connector end connects to a disk or a SS port on a controller. The SS cable QSFP connectors are keyed; when oriented correctly into a SS port, the QSFP connector clicks into place and the disk SS port LNK LE illuminates green. o not force a connector into a port. The following illustration shows a QSFP connector: Note: The QSFP connector shown has a pull tab for releasing the connector from a SS port; however, QSFP connectors can have a different release mechanism. v isk shelves connected with SS optical cables require a version of disk firmware that supports SS optical cables. est practice is to update all disk shelves in the storage system with the latest version of disk firmware. Note: o not revert disk firmware to a version that does not support SS optical cables. 2 IM System Storage N series: Universal SS and P abling Guide
v SS optical multimode QSFP-to-QSFP cables can be used for controller-to- and -to- connections, and are available in lengths up to 50 meters. v If you are using SS optical multimode MPO cables with MPO QSFP modules, the following parameters apply: You can use these cables for controller-to- and -to- connections. The length of a single cable cannot exceed 150 meters for OM4 and 100 meters for OM3. The total end-to-end path (sum of point-to-point paths from the controller to the last ) cannot exceed 510 meters. The total path includes the set of breakout cables, patch panels, and inter-panel cables. v If you are using SS optical multimode breakout cables, the following parameters apply: You can use these cables for controller-to- and -to- connections. If you use multimode breakout cables for a -to- connection, you can only use it once within a stack of disk shelves. You must use SS optical multimode QSFP-to-QSFP or MPO cables with MPO QSFP modules to connect the remaining -to- connections. The point-to-point (QSFP-to-QSFP) path of any multimode cable cannot exceed 150 meters for OM4 and 100 meters for OM3. The path includes the set of breakout cables, patch panels, and inter-panel cables. The total end-to-end path (sum of point-to-point paths from the controller to the last ) cannot exceed 510 meters. The total path includes the set of breakout cables, patch panels, and inter-panel cables. Up to one pair of patch panels can be used in a path. You need to supply the patch panels and inter-panel cables. The inter-panel cables must be the same mode as the SS optical breakout cable: multimode. You received a set of QSFP-to-MPO cable modules with each set of SS optical breakout cables, which you must attach to the MPO end of each SS optical breakout cable. The breakout cables have S, L, or MTRJ connectors on the opposite end, which connect to a patch panel. You must connect all eight (four pairs) of the S, L, or MTRJ breakout connectors to the patch panel. SS cabling for new storage systems 3
v QSFP-to-mini-SS copper cables are used for N3600 configurations only, and connect disk shelves to the dual-port mini-ss Hs on the controller. The following illustration shows the QSFP-to-mini-SS cable connectors: v SS ports on the disk are designated by a square or circle symbol. IOM P SS Square ports are always cabled to circle ports, and circle ports are always cabled to square ports. Never cable square ports to square ports or circle ports to circle ports. The following illustration shows square and circle port labeling on an IOM. Onboard SS port rules Storage systems that have controllers with onboard SS ports must follow a specific port identification rule. For all systems except N3400, an onboard H is defined as occupying PI slot 0, with each port named with sequentially increasing letters (0a, 0b, 0c, and so on). This is to ensure that the SS cabling rules apply to all SS Hs on a controller, whether in a physical PI slot (slot 1-N) or on board the base controller. The following example shows a controller with one onboard SS H in slot 0 and three SS Hs in physical PI slots 1, 2, and 3. 4 IM System Storage N series: Universal SS and P abling Guide
Slot 0 Slot 2 Slot 3 For N3400 systems, the onboard SS port labeled 0d should be treated as if it were slot 0 port a. Treating the port as 0a enables you to apply all the SS cabling rules. (ata ONTP continues to recognize this port as 0d.) The following example shows the N3400 exception: 2040 controller 0a 0d Shelf-to- connection rules When you have more than one disk in a stack of disk shelves, the shelves in the stack must be connected to each other (daisy-chained). The disk shelves are daisy-chained in the following way: v IOM circle port is connected to the next IOM square port. v IOM circle port is connected to the next IOM square port. Note: Square ports are always cabled to circle ports, and circle ports are always cabled to square ports. Never cable square ports to square ports or circle ports to circle ports. The following illustration shows a daisy-chained stack of disk shelves with the IOMs arranged side-by-side, similar to how an EXN3500 disk is oriented. SS cabling for new storage systems 5
Stack of three shelves P SS P SS IOM IOM The following illustration shows a daisy-chained stack of disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Stack of three shelves P SS IOM IOM ontroller-to--connection rules Each controller connects to the first and last disk in each stack in the system through the controller SS ports. The controller SS and ports connect to the first disk square ports. The controller SS and ports connect to the last disk circle ports. These controller SS ports can be on a SS H in a physical PI slot (slot 1-N) or on the base controller. The SS cabling algorithm described in this document balances SS cabling between the onboard SS ports (as applicable) and SS H ports to provide highly available storage-controller configurations. This cabling algorithm meets the following goals: v Provides a single, easily understood universal algorithm for all SS products and configurations. v Is verifiable by configuration-checking software and tools. 6 IM System Storage N series: Universal SS and P abling Guide
v Provides maximum possible resilience to maintain availability and minimize the reliance on controller takeovers. ny deviations from the algorithm are not recommended; deviations may reduce reliability, universality, and commonality. Note: If your system cabling is not compliant with the rules in this guide, it is recommended that you correct the cabling during a maintenance window. and port connections v ontroller SS ports and are connected to the first disk- square port in each stack of disk shelves. The and ports on controller 1/ connect to IOM. The and ports on controller 2/ connect to IOM. For example, the following illustration shows how controller ports and are connected to disk shelves that have the IOMs arranged side by side, similar to how a EXN3500 disk is oriented: ontroller 1/ ontroller 2/ Stack 1 Stack 2 P SS P SS P SS P SS IOM IOM IOM IOM For example, the following illustration shows how controller ports and are connected to disk shelves that have the IOMs arranged one above the other, similar to how a EXN3000 disk is oriented: SS cabling for new storage systems 7
ontroller 1/ ontroller 2/ Stack 1 P SS Stack 2 P SS IOM IOM v ontroller SS ports and are connected to the disk- stacks in sequential order of the PI slots (slot 0, slot 1, slot 2, and so on) on the controller. For example, a controller with slots 1, 2, and 3 occupied with quad-port SS Hs is cabled in order of slots and ports 1a, 2a, 3a, 1c, 2c, 3c: ontroller Slot 2 Slot 3 For example, a controller with slots 0, 1, and 2 occupied (slot 0 being an onboard port and slot 1 and slot 2 being quad-port SS Hs) is cabled in order of slots and ports 0a, 1a, 2a, 1c, 2c: ontroller Slot 0 Slot 2 8 IM System Storage N series: Universal SS and P abling Guide
and port connections v ontroller SS ports and are connected to the last disk- circle port in each stack of disk shelves. The and ports on controller 1/ connect to IOM. The and ports on controller 2/ connect to IOM. For example, the following illustration shows how controller ports and are connected to disk shelves that have the IOMs arranged side by side, similar to how a EXN3500 disk is oriented: ontroller 1/ ontroller 2/ Stack 1 Stack 2 P SS P SS P SS P SS IOM IOM IOM IOM For example, the following illustration shows how controller ports and are connected to disk shelves that have the IOMs arranged one above the other, similar to how a EXN3000 disk is oriented: SS cabling for new storage systems 9
ontroller 1/ ontroller 2/ Stack 1 P SS IOM IOM Stack 2 P SS v ontroller SS ports and are connected to the disk- stacks by offsetting the order of the PI slots by one so that the first port on the first slot is cabled last. The cabling offset ensures system resiliency. For example, a controller with three quad-port SS Hs in slots 1, 2, and 3 is cabled in order of slots and ports 2b, 3b, 1d, 2d, 3d, 1b: ontroller Slot 2 Slot 3 For example, a controller with slots 0, 1, and 2 occupied (slot 0 being an onboard port and slot 1 and 2 being quad-port SS Hs) is cabled in order of slots and ports 1b, 2b, 1d, 2d, 0b: 10 IM System Storage N series: Universal SS and P abling Guide
ontroller Slot 0 Slot 2 controller with one quad-port H is cabled in order of port 1d and then 1b: ontroller Stack cabling sequence ontrollers are cabled to the stacks according to the sequence of the stacks, such as stack 1, stack 2, stack 3, and so on. N3400 configurations For single-controller N3400 configurations, the controller is in Slot (the bottom slot); therefore, the controller is cabled as if it were controller 2/. If a second controller is added to make this an H pair configuration, the new controller is cabled as controller 1/. ontroller-to-controller rule (N3240 systems only) On N3240 systems in H pairs with no external storage, external cables should connect the SS ports on the two controller modules to enable multipath H for the internal disks. abling SS connections abling the SS connections involves completing the SS cabling worksheet to identify the SS ports that need to be cabled to the disk shelves, and applying the worksheet when following the cabling procedure. SS cabling for new storage systems 11
ompleting the SS cabling worksheet y using the SS cabling worksheet to list the slots and SS ports for each controller, you can simplify the cabling task. bout this task You can reference the appropriate Sample SS cabling worksheet when recording your information. Procedure 1. Print out a copy of the SS cabling worksheet template to record your information. 2. Read the cabling worksheet rules and be sure to understand them. 3. Make a list of all SS ports and then all SS ports on each controller in your configuration. controller with three quad-port SS Hs is listed as 1a, 2a, 3a, 1c, 2c, 3c. controller with one quad-port SS H is listed as 1a, 1c. N3600 with one dual-port SS H is listed as 1a. N3400 is listed as 0a. N6200 series dual enclosure (with two onboard SS ports and one quad-port) is listed as 0a, 1a, 1c. 4. If you have a N3400 go to Step 5; otherwise, list the remaining ports by completing the following substeps: a. Make a list of all SS ports and then all SS ports on each controller in your configuration (in the grayed-out cells of the worksheet). controller with three quad-port SS Hs is listed as 1b, 2b, 3b, 1d, 2d, 3d. controller with one quad-port SS H is listed as 1b, 1d. N3600 with one dual-port SS H is listed as 1b. N6200 series dual enclosure (with two onboard SS ports and one quad-port) is listed as 0b, 1b, 1d. b. Rewrite the list (on the line below the grayed-out cells) so that the first 1b, 2b, 3b, 1d, 2d, 3d, 1b port in your list is moved to the end of your list: Offsetting the list by one (so that the cabling begins on the second slot/h) ensures system resiliency when cabling your storage system. controller with three quad-port SS Hs is listed as 2b, 3b, 1d, 2d, 3d, 1b. controller with one quad-port SS H is listed as 1d, 1b. N3600 with one dual-port SS H is listed as 1b. N6200 series dual enclosure (with two onboard SS ports and one quad-port) is listed as 1b, 1d, 0b. X 12 IM System Storage N series: Universal SS and P abling Guide
c. ross out the information in the grayed-out cells. You don't need this information anymore. You will use the offset list below the grayed-out cells to cable your system. 5. Go to abling SS ports to cable your storage system. SS cabling worksheet rules and template y understanding the rules for filling out the worksheet template, you can record your controller SS slot and port information correctly. Use the following rules when recording your information in the SS cabling worksheet. Samples of completed worksheets are shown in the Sample cabling worksheet sections. v Ports are listed in sequence of the slot numbers to which they belong and include the slot number. For example, onboard SS ports belong to slot 0. SS port of slot 0 is written as 0a. single quad-port SS H in a physical PI slot belongs to slot 1. dditional quad-port SS Hs belong to slot 2, slot 3, and so on. SS port of slot 1 is written as 1a. SS port of slot 2 is written as 2a, and so on. v If your controller has onboard ports, they are always listed first (since they belong to slot 0) and then followed by the quad-port SS H slots (slot 1, 2, and so on). v If you have an H pair, the port list is the same for controller 1/ and controller 2/. v Only one controller slot and SS port is listed in a table cell under a stack number. v ontroller slots and SS ports are listed in sequence of the stack numbers. Note: This worksheet provides space for eight stacks. o not skip any table cells. v Only fill in controller slots and SS ports for the number of stacks you have in your configuration. For example, if you have only four stacks of disk shelves in your configuration, the columns for stack numbers 5 through 8 remain blank. v The controller SS ports and rows show two lines for controller 1/ and controller 2/. Use the first line (grayed-out) for recording the port list prior to offsetting the list by one. Use the second line for recording the list with the ports offset by one. SS cabling for new storage systems 13
v The worksheet has you connect square ports first, but you can cable circle ports first. However, square ports must only be connected to circle ports and circle ports must only be connected to square ports. v If your configuration has a single stack, the references to the first and last apply to the one. v If your configuration has a N3400 with a single controller, list the slot and port information on the ontroller 2/ line. Single controller N3400 systems have the controller in the bottom (2/) slot of the chassis. ontroller SS ports and ontroller ontroller 1/ ontroller 2/ ontroller 1/ ontroller SS ports and stack cabling sequence Stack number (1-8) 1 2 3 4 5 6 7 8 able to disk IOM To first IOM To first IOM To last IOM and ontroller 2/ To last IOM Sample SS cabling worksheet for a N6200 series You can reference this sample SS cabling worksheet when filling in your own worksheet. The sample worksheet shows a N6200 series dual enclosure configuration (two onboard SS ports and one quad-port SS H) with two stacks of disk shelves. Note: The slots and SS ports shown in gray text for stack 3 are unused because of the limited number of stacks in this configuration. If there were three stacks, all three slots and SS ports listed would be used. Note: The two rows which are grayed-out and crossed-out are the original port lists that were made prior to offsetting the lists by one. The lines below that show the ports offset by one and would be used to cable the system. 14 IM System Storage N series: Universal SS and P abling Guide
ontroller SS ports and and ontroller ontroller 1/ ontroller 2/ ontroller 1/ ontroller 2/ ontroller SS ports and stack cabling sequence Stack number (1-8) 1 2 3 4 5 6 7 8 0a 1a 1c 0a 1a 1c 0b 1b 0b 1b 1b 1d 1b 1d 1d 0b 1d 0b able to disk IOM To first IOM To first IOM To last IOM To last IOM The following illustrates a cabled N6200 series using the information in the sample worksheet. Two examples are shown. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. 1. ontroller 1/ Slot 0 Multipath H configuration 5. 2. 6. 3. ontroller 2/ Slot 0 Stack 1 Stack 2 P SS P SS P SS P SS IOM IOM IOM IOM 4. 7. 8. SS cabling for new storage systems 15
1. ontroller 1/ ontroller 2/ Slot 0 Multipath H onfiguration Slot 0 2. 3. 4. 5. Stack 1 6. Stack 2 7. 8. P SS P SS IOM IOM Sample SS cabling worksheet for an H pair (non N6200 series, N3240, or N3400 ) You can reference this sample SS cabling worksheet and illustration when filling in your own worksheet. The sample worksheet shows an H pair configuration with three quad-port SS Hs and three stacks of disk shelves. Note: The slots and SS ports shown in gray text for stacks 4, 5, and 6 are unused because of the limited number of stacks in this configuration. If there were six stacks, all six slots and SS ports listed would be used. Note: The two rows which are grayed-out and crossed-out are the original port lists made prior to offsetting the lists by one. The lines below that show the ports offset by one and would be used to cable the system. 16 IM System Storage N series: Universal SS and P abling Guide
ontroller SS ports and and ontroller ontroller 1/ ontroller 2/ ontroller 1/ ontroller 2/ ontroller SS ports and stack cabling sequence Stack number (1-8) 1 2 3 4 5 6 7 8 1a 2a 3a 1c 2c 3c 1a 2a 3a 1c 2c 3c 1b 2b 3b 1d 2d 3d 2b 3b 1d 2d 3d 1b 1b 2b 3b 1d 2d 3d 2b 3b 1d 2d 3d 1b able to disk IOM To first IOM To first IOM To last IOM To last IOM The following illustrates a cabled H pair (non N6200 series, N3240, or N3400 ) using the information in the sample worksheet. Two examples are shown. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Multipath H configuration ontroller 1/ ontroller 2/ 1. Slot 2 Slot 3 Slot 2 Slot 3 6. 4. 9. 8. 11. 12. 7. 3. 2. Stack 1 Stack 2 Stack 3 5. 10. P SS P SS P SS P SS P SS P SS IOM IOM IOM IOM IOM IOM SS cabling for new storage systems 17
Multipath H configuration ontroller 1/ ontroller 2/ 1. Slot 2 Slot 3 4. Slot 2 Slot 3 6. 9. 7. 8. 3. 11. Stack 1 2. Stack 2 12. Stack 3 IOM IOM P SS P SS 5. P SS 10. abling SS ports abling the SS ports involves cabling the -to- connections and the -to-controller connections from the first and last in a stack, then verifying that the SS connections are correct. efore you begin v You should have filled out the SS cabling worksheet. v Your system should have met the disk and SS cable requirements as stated in the IM N series Introduction and Planning Guide. v Your storage system's controllers and disk shelves must already be installed in a rack if the system was not shipped in a system cabinet. v Your storage system must already have PIe SS Hs installed and available for cabling the disk shelves if the system will use one or more PIe SS Hs. bout this task v SS cables with QSFP connectors are used to connect disk shelves together and to connect disk shelves to the SS ports on the controller. The SS cable QSFP connectors are keyed; when oriented correctly into a SS port, the QSFP connector clicks into place and the disk SS port LNK LE illuminates green. o not force a connector into a port. 18 IM System Storage N series: Universal SS and P abling Guide
v The SS cables can be SS copper, SS optical, or a mix depending on whether or not your system meets the requirements for using the type of cable. If you are using a mix of SS copper cables and SS optical cables, the following rules apply: Shelf-to- connections in a stack must be all SS copper cables or all SS optical cables. If the -to- connections are SS optical cables, the -to-controller connections to that stack must also be SS optical cables. If the -to- connections are SS copper cables, the -to-controller connections to that stack can be SS optical cables or SS copper cables. v If you are using SS optical multimode breakout cables, the following parameters apply: You can use these cables for controller-to- and -to- connections. If you use multimode breakout cables for a -to- connection, you can only use it once within a stack of disk shelves. You must use SS optical multimode QSFP-to-QSFP or MPO cables with MPO QSFP modules to connect the remaining -to- connections. The point-to-point (QSFP-to-QSFP) path of any multimode cable cannot exceed 150 meters for OM4 and 100 meters for OM3. The path includes the set of breakout cables, patch panels, and inter-panel cables. The total end-to-end path (sum of point-to-point paths from the controller to the last ) cannot exceed 510 meters. The total path includes the set of breakout cables, patch panels, and inter-panel cables. Up to one pair of patch panels can be used in a path. You need to supply the patch panels and inter-panel cables. The inter-panel cables must be the same mode as the SS optical breakout cable: multimode. You received a set of QSFP-to-MPO cable modules with each set of SS optical breakout cables, which you must attach to the MPO end of each SS optical breakout cable. The breakout cables have S, L, or MTRJ connectors on the opposite end, which connect to a patch panel. You must connect all eight (four pairs) of the S, L, or MTRJ breakout connectors to the patch panel. SS cabling for new storage systems 19
v isk shelves connected with SS optical cables require a version of disk firmware that supports SS optical cables. est practice is to update all disk shelves in the storage system with the latest version of disk firmware. Note: o not revert disk firmware to a version that does not support SS optical cables. Procedure 1. able the -to- SS connections by completing the following substeps if you have more than one disk in a stack; otherwise, go to Step 2: a. onnect IOM circle port to the next IOM square port. b. onnect IOM circle port to the next IOM square port. The following examples show a stack of three shelves connected -to-. The first example shows shelves that have the IOMs arranged side-by-side, similar to how an EXN3500 disk is oriented. The second example shows shelves that have the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Stack of three shelves P SS P SS IOM IOM 20 IM System Storage N series: Universal SS and P abling Guide
Stack of three shelves P SS IOM IOM 2. Use your completed worksheet to cable the SS and SS connections for each controller. Note: If you are cabling a N3600 storage system, use the QSFP to mini-ss cables for all controller-to- connections; otherwise, use supported QSFP to QSFP SS cables. 3. Use your completed worksheet to cable the SS and SS connections for each controller. Note: For quad-port SS Hs, ports and never connect to the same stack of disk shelves and ports and never connect to the same stack of disk shelves. This means that you might have unused H ports, depending on the number of stacks in your configuration. For example, a single controller with one quad-port SS H and one stack of shelves only uses H ports and. 4. Verify the SS connections by entering the applicable command: v For 7-Mode, enter the following command at the system console: sasadmin expander_map For H pairs, you can run this command on either node. v For clustered systems, enter the following command at the clustershell prompt: run -node node_name -command sasadmin expander_map The output shows information about SS expanders attached to SS channels on the controllers. For example, the following output shows the number one expander in Slot is connected to channel 4a on the controller. Expanders on channel 4a: Level 3: WWN 500a0980000840ff, I 1, Serial Number ' SHU0954292G114', Product 'S424IOM6 ', Rev '0151', Slot SS cabling for new storage systems 21
5. The next step depends on the output. If... Then... The output lists all IOMs The IOMs have connectivity; go to Step 6. ny IOMs are not shown (either the output does not show an IOM because it is cabled incorrectly, or the output does not show all the IOMs downstream from the incorrectly cabled IOM.) Repeat Steps 1 through 5 to correct cabling errors, then go to Step 6. 6. If you are using SS optical cables, verify that the all disk shelves in the storage system have the latest version of disk firmware by completing the following substeps; otherwise, go to Step 8. a. Enter the following command at the storage system console: sasadmin expander_map b. Locate the disk firmware information for the disk shelves in the output. 0151 is the disk firmware version for number one (Slot /IOM ) in the storage system: Expanders on channel 4a: Level 3: WWN 500a0980000840ff, I 1, Serial Number ' SHU0954292G114', Product 'S424IOM6 ', Rev '0151', Slot c. ompare the firmware information in the command output with the disk firmware information at www.ibm.com/storage/support/ nseries to determine the most current disk firmware version. 7. The next step depends on how current the disk firmware is. If the firmware version in the command output is... The same or later than the most current version on the N series support website (accessed and navigated as described in Websites) n earlier version than the most current version on the N series support website (accessed and navigated as described in Websites) Then... omplete the remaining steps in this procedure. No disk firmware update is needed. Your system is running a version of disk firmware that supports SS optical cables. omplete the remaining steps in this procedure and download the disk firmware when instructed. 8. heck whether your system has disk autoassignment enabled by using the options disk command at the console of either controller. If disk autoassignment is enabled, the output shows disk.auto_assign on. 22 IM System Storage N series: Universal SS and P abling Guide
9. If your system does not have disk autoassignment enabled or disk drives in the same stack are owned by both controllers, assign disk drive ownership. The following substeps are for basic disk ownership assignment for 7-Mode. To specify the disk drives to be assigned or the system to own the disk drives, see the ata ONTP Storage Management Guide for 7-Mode. For 7-Mode H pairs, you can run these commands from the system console of either node. For clustered systems, see the ata ONTP luster-mode Storage and ata Protection Management Guide for disk drive ownership information on the N series support website (accessed and navigated as described in Websites). a. Enter the following command to see all of the unassigned disk drives: disk show -n The disk drives in the hot-added have the ownership state of Not Owned. b. Enter the following command to assign ownership of the disk drives: disk assign all If you need to assign the disk drives to a non-local storage system, you can specify a storage system by using the -o owner_name option. To determine the owner_name, you can run the hostname command at the console of either controller. c. Enter the following command to verify the assignments that you made: disk show -v 10. The next step depends on whether you are using the P capability. If you are... Using the P capability Then... Go to abling P connections. Note: If in Step 7, you had an earlier version of disk firmware than the most current version on the N series support website (accessed and navigated as described in Websites), update the firmware as instructed when cabling the P connections. SS cabling for new storage systems 23
If you are... Not using the P capability Then... 1. oot the storage system and begin setup. 2. If in Step 7, you had an earlier version of disk firmware than the most current version on the N series support website (accessed and navigated as described in Websites), download the disk firmware file by using the procedure at www.ibm.com/storage/support/ nseries. For H pairs, you can run the commands from either node. SS cabling examples You can use the SS cabling examples to see how the SS cabling rules are applied for common configurations. Examples of SS cabling for H pairs with quad-port Hs You can reference these examples for multipath H configurations with quad-port SS Hs. The numbered cables show the sequence in which the system should be cabled. Two examples are shown for each configuration type. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Note: These examples do not apply to N3400, N3240, or N6200 series systems. 24 IM System Storage N series: Universal SS and P abling Guide
One quad-port SS H and a single stack The SS port cabling sequence for each controller is 1a, 1d. Multipath H configuration ontroller 1/ ontroller 2/ 1. 2. Ethernet port Ethernet port 4. Stack 1 3. P SS P SS IOM IOM Multipath H configuration 1. ontroller 1/ Ethernet port ontroller 2/ Ethernet port 4. 2. P SS 3. IOM IOM SS cabling for new storage systems 25
One quad-port SS H and two single stacks The SS port cabling sequence for each controller is 1a, 1c, 1d, 1b. Multipath H configuration ontroller 1/ 5. 2. ontroller 1/ 7. 3. 8. 4. 1. 6. P SS P SS P SS P SS IOM IOM IOM IOM Oultipath H configuration ontroller 1/ 2. 3. 5. 6. ontroller 1/ 8. 4. 7. 1. M SS M SS PI O PI O 26 IM System Storage N series: Universal SS and P abling Guide
Two quad-port SS Hs and two stacks of disk shelves The SS port cabling sequence for each controller is 1a, 2a, 2b, 1d. Multipath H configuration ontroller 1/ ontroller 2/ Slot 2 1. 2. Ethernet port 3. Slot 2 Ethernet port 4. 8. 7. Stack 1 Stack 2 P SS P SS P SS P SS 5. IOM IOM IOM IOM 6. SS cabling for new storage systems 27
Multipath H onfiguration ontroller 1/ ontroller 2/ Slot 2 2. Slot 2 4. 1. Ethernet port 3. Ethernet port 7. Stack 1 Stack 2 5. P SS P SS 6. IOM IOM 8. Three quad-port SS Hs and three stacks of disk shelves The SS port cabling sequence for each controller is 1a, 2a, 3a, 2b, 3b, 1d. Multipath H configuration ontroller 1/ ontroller 2/ 1. Slot 2 Slot 3 Slot 2 Slot 3 6. 4. 9. 8. 11. 12. 7. 3. 2. Stack 1 Stack 2 Stack 3 5. 10. P SS P SS P SS P SS P SS P SS IOM IOM IOM IOM IOM IOM 28 IM System Storage N series: Universal SS and P abling Guide
Multipath H configuration ontroller 1/ ontroller 2/ 1. Slot 2 Slot 3 4. Slot 2 Slot 3 6. 9. 7. 8. 3. 11. Stack 1 2. Stack 2 12. Stack 3 IOM IOM P SS P SS 5. P SS 10. Four quad-port SS Hs and four stacks of disk shelves The SS port cabling sequence for each controller is 1a, 2a, 3a, 4a, 2b, 3b, 4b, 1d. Four quad-port SS Hs and eight stacks of disk shelves The SS port cabling sequence for each controller is 1a, 2a, 3a, 4a, 1c, 2c, 3c, 4c, 2b, 3b, 4b, 1d, 2d, 3d, 4d, 1b. Five quad-port SS Hs and five stacks of disk shelves The SS port cabling sequence for each controller is 1a, 2a, 3a, 4a, 5a, 2b, 3b, 4b, 5b, 1d, 2d, 3d, 4d, 1b. Five quad-port SS Hs and 10 stacks of disk shelves The SS port cabling sequence for each controller is 1a, 2a, 3a, 4a, 5a, 1c, 2c, 3c, 4c, 5c, 2b, 3b, 4b, 5b,1d, 2d, 3d, 4d, 5d,1b. SS cabling for new storage systems 29
Examples of SS cabling for single-controller configurations with quad-port Hs You can reference these SS cabling examples for single-controller configurations with quad-port SS Hs (dual-path configurations). The numbered cables show the sequence in which the system should be cabled. Two examples are shown for each configuration type. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Note: These examples do not apply to N3400, N3240, or N6200 series systems. One quad-port SS H and one stack of disk shelves The SS port cabling sequence is 1a, 1d. ual-path configuration ontroller 1/ 1. Stack 1 2. P SS P SS IOM IOM 30 IM System Storage N series: Universal SS and P abling Guide
ual-path configuration 1. IOM IOM ontroller 1/ Stack 1 P SS 2. One quad-port SS H and two stacks of disk shelves The SS port cabling sequence is 1a, 1c,1d, 1b. ual-path configurations 3. 1. ontroller 1/ 4. 2. Stack 1 Stack 2 P SS P SS P SS P SS IOM IOM IOM IOM SS cabling for new storage systems 31
ual-path configuration 1. ontroller 1/ 3. 2. 4. Stack 1 Stack 2 P SS P SS IOM IOM Two quad-port SS Hs and two stacks of disk shelves The SS port cabling sequence is 1a, 2a, 2b, 1d. 4. ual-path configurations 1. ontroller 1/ Slot 2 Ethernet port 2. 3. Stack 1 Stack 2 P SS P SS P SS P SS IOM IOM IOM IOM 32 IM System Storage N series: Universal SS and P abling Guide
ual-path configuration ontroller 1/ Slot 2 1. Ethernet port 2. 3. 4. Stack 1 Stack 2 P SS P SS IOM IOM Examples of SS cabling for N6200 series systems You can reference these SS cabling examples for N6200 series dual enclosure H and single enclosure H configurations. The numbered cables show the sequence in which the system should be cabled. Two examples are shown for each configuration type. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Note: You can also view the N6200 Series System abling Examples document a more real-world cabling example of a N6200 series system installed in a cabinet on the N series support website (accessed and navigated as described in Websites) at www.ibm.com/storage/support/ nseries. SS cabling for new storage systems 33
N6200 series dual enclosure H configuration The SS port cabling sequence for each controller is 0a,1a,1b,1d. 1. ontroller 1/ Slot 0 Multipath H configuration 5. 2. 6. 3. ontroller 2/ Slot 0 Stack 1 Stack 2 P SS P SS P SS P SS IOM IOM IOM IOM 4. 7. 8. 1. ontroller 1/ ontroller 2/ Slot 0 Multipath H onfiguration Slot 0 2. 3. 4. 5. Stack 1 6. Stack 2 7. 8. P SS P SS IOM IOM 34 IM System Storage N series: Universal SS and P abling Guide
N6200 series single enclosure H configuration The SS port cabling sequence for each controller is 0a, 0b. Multipath H configuration 3. 1. Slot 0 Slot 0 Stack 1 P SS P SS ontroller 1/ ontroller 2/ 2. 4. IOM IOM Multipath H configuration Slot 0 1. ontroller 1/ 3. Slot 0 2. ontroller 2/ 4. Stack 1 P SS IOM IOM SS cabling for new storage systems 35
Examples of SS cabling for PM systems You can reference these SS cabling examples for H pair and single-controller configurations. The numbered cables show the sequence in which the Platform ontroller Module (PM) system should be cabled. For configurations with external disk shelves, two examples are shown for each configuration type. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 is oriented. Note: The examples show a system with PMs arranged side by side, similar to how a N3150 is oriented, but the cabling applies to systems with PMs arranged one above the other, similar to how an3240 is oriented. H pair configuration with one stack of disk shelves (multipath H) The SS port cabling sequence for each controller is 0a, 0b. PM1 PM2 SS P SS P 0b 0a 0b 0a 1. PSU 2. PSU 3. P SS P SS IOM IOM 4. P SS P SS P SS P SS 36 IM System Storage N series: Universal SS and P abling Guide
PM1 PM2 SS P SS P 0b 0a 0b 0a PSU PSU 1. 2. P SS 3. IOM IOM 4. Single-controller configuration with one stack of disk shelves (dual-path) The SS port cabling sequence is 0a, 0b. Note: ual-path provides greater resiliency, but when dual-path is used, ata ONTP 8.1.x issues occasional warnings to the console because mixed paths are detected. To avoid the warnings, single-path connections to the external SS storage are a supported option for these configurations. SS cabling for new storage systems 37
SS P 0b 0a PSU 1. P SS P SS IOM IOM PSU 2. P SS P SS P SS P SS SS P 0b 0a PSU PSU P SS IOM IOM 38 IM System Storage N series: Universal SS and P abling Guide
Single-controller configuration with one stack of disk shelves (single-path) The SS port cabling sequence is 0a. SS P 0b 0a PSU PSU P SS P SS IOM IOM P SS P SS P SS P SS SS cabling for new storage systems 39
SS P 0b 0a PSU PSU P SS IOM IOM H pair configuration with no external disk shelves (multipath H) The SS port cabling sequence for each controller is 0a, 0b. PM1 PM2 SS P SS P 0b 0a 0b 0a PSU PSU Examples of SS cabling for N3600 systems You can reference these SS cabling examples for N3600 H pair (multipath H) and single-controller (dual-path) configurations. The numbered cables show the sequence in which the system should be cabled. 40 IM System Storage N series: Universal SS and P abling Guide
Two examples are shown for each configuration type. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Note: For single-controller N3400 storage systems, the controller is in Slot (the bottom slot); therefore, the controller is cabled as if it were controller 2/. If a second controller is added to make this an H pair configuration, the new controller is cabled as controller 1/. Note: ecause N3600 systems only have one dual-port SS H on each controller, a maximum of one stack of disk shelves is supported in a multipath H or dual-path configuration. N3600 multipath H configuration The SS port cabling sequence for each controller is 1a, 1b. ontroller 1/ H pair configuration ontroller 2/ 1. Ethernet port 2. Ethernet port Stack 1 4. 3. P SS P SS IOM IOM SS cabling for new storage systems 41
ontroller 1/ H pair configuration ontroller 2/ 1. Ethernet port 2. Ethernet port Stack 1 P SS 3. IOM IOM 4. N3600 multipath H configuration with one disk The SS port cabling sequence for each controller is 1a, 1b. 1. H pair configuration ontroller 1/ ontroller 2/ Ethernet port Ethernet port 4. Stack 1 2. 3. P SS P SS IOM IOM 42 IM System Storage N series: Universal SS and P abling Guide
1. H pair configuration ontroller 1/ ontroller 2/ Ethernet port Ethernet port 3. IOM P SS 4. 2. IOM N3600 dual-path configuration The SS port cabling sequence is 1a, 1b. Single-controller configuration ontroller 2/ 1. 2. Stack 1 P SS P SS IOM IOM SS cabling for new storage systems 43
Single-controller configuration 1. ontroller 2/ 2. IOM IOM Stack 1 P SS N3600 dual-path configuration with one disk The SS port cabling sequence is 1a,1b. Single-controller configuration ontroller 2/ Ethernet port 2. 1. Stack 1 P SS P SS IOM IOM 44 IM System Storage N series: Universal SS and P abling Guide
Single-controller configuration ontroller 2/ 1. Ethernet port 2. P SS IOM IOM Examples of SS cabling for N3400 systems You can reference these SS cabling examples for N3400 H pair (single-path H) and single-controller (single-path) configurations. The numbered cables show the sequence in which the system should be cabled. Two examples are shown for each configuration type. One example shows disk shelves with the IOMs arranged side by side, similar to how an EXN3500 disk is oriented. The second example shows disk shelves with the IOMs arranged one above the other, similar to how an EXN3000 disk is oriented. Note: For single-controller N3400 storage systems, the controller is in Slot (the bottom slot); therefore, the controller is cabled as if it were controller 2/. If a second controller is added to make this an H pair configuration, the new controller is cabled as controller 1/. Note: The N3400 storage system s onboard SS port labeled 0d is treated as slot 0 port a (0a) for cabling purposes. When the onboard port is treated as port 0a, all the SS cabling rules can be applied. Note: ecause N3400 systems only have one SS port on each controller, a maximum of one stack of disk shelves is supported. SS cabling for new storage systems 45