Why You Should Deploy Switched-FICON. David Lytle, BCAF Global Solutions Architect System z Technologies and Solutions Brocade

Transcription

1 Why You Should Deploy Switched-FICON David Lytle, BCAF Global Solutions Architect System z Technologies and Solutions Brocade

2 Legal Disclaimer All or some of the products detailed in this presentation may still be under development and certain specifications, including but not limited to, release dates, prices, and product features, may change. The products may not function as intended and a production version of the products may never be released. Even if a production version is released, it may be materially different from the pre-release version discussed in this presentation. NOTHING IN THIS PRESENTATION SHALL BE DEEMED TO CREATE A WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT OF THIRD-PARTY RIGHTS WITH RESPECT TO ANY PRODUCTS AND SERVICES REFERENCED HEREIN. Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing symbol, DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc. or its subsidiaries, in the United States and/or in other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. 2

3 Switched-FICON is a Best Practice for System z Brocade FICON switching devices do not cause performance problems within a local data center Architected and deployed correctly, Brocade FICON switching devices do not cause performance problems even across very long distances In fact, use of Brocade switched-ficon and Brocade FCIP long distance connectivity solutions can even enhance DASD replication performance and long distance tape operations effectiveness and performance Switched-FICON is the only way to efficiently and effectively support Linux on System z connectivity Switched-FICON is the only way to really take advantage of the full value of the System z I/O subsystem Over the next set of slides you ll discover why this is true 3

4 SM to MM Conversion and Vice Versa Point-to-Point Deployment of FICON Long-wave Optics FICON Point-to-Point Would have to be long-wave Optics Longwave-to-Shortwave Long-wave Optics Long-wave FICON DCX Long-wave to short-wave conversion without having to modify storage optics Short-wave or long-wave Optics Short-wave Most IBM FICON Express8 channel cards have long-wave optics IBM is pushing long-wave in the data center They know that 16 Gbps is on the near horizon If your on the floor legacy storage currently has short-wave ports: End-to-end connections must be the same LW-to-LW SW-to-SW You would have to be sure that storage ports were also long-wave potential budget hit for optics! Long-wave optics for storage and Directors is more expensive than short-wave optics Switched FICON allows long-wave into the switch/director and short-wave out of the switch/director Port by port basis in the switching device Just have to order the proper port cards Financially better FICON TCO 4

5 Point-to-Point versus Switched-FICON Point-to-point FICON is exactly the same as what is called in the Open Systems world - Direct Attach Storage (DAS) And it also suffers from exactly the same issues that caused companies to being implementing SANs in their growing storage environments beginning in the late 1990 s Switched-FICON provides very compelling benefits to the user in the areas of: Consolidation (dynamic connections to fewer storage boxes) High Availability (far beyond point-to-point and over distance) Management (especially of fast or abrupt growth - planning) Performance (balanced and better use of all channel resources) Scalability (in many different ways including Fan In-Fan out) Let s discuss this in detail on the following slides 5

6 Point-to-Point FICON is almost dead! System z There are a LOT of reasons for not getting caught in the trap of doing direct-attached FICON: Maximum Point-to-Point distances are going to shrink dramatically as bandwidth increases Reliability Scalability As of FICON Express8, buffer credits are also shrinking 6

7 End-to-End FICON/FCP Connectivity System z Fiber Cable Cabling Considerations Long wave single mode still works well 1/2/4/8/10 Gbps out to 10km with SM Short wave multi-mode might be limiting 4G optics auto-negotiate back to 1, 2G 8G optics auto-negotiate back to 2, 4G 1G storage connectivity requires 2/4G SFPs Distance with Multi-Mode Cables (meters) 7

8 End-to-End FICON/FCP Connectivity System z cable distances at 8Gbps OS1 9m OM3 50m 150 meters 492 feet 2, 4 and 8Gbps 4 to 10 km 2.5 to 6.2 miles System z OM2 50m 50 meters 164 feet Longwave 8G on a z10 only comes at 10km no 4km ~80% of System z CHPIDs are long wave But older OM2 cables are going to start requiring the mainframe PtP devices to hover very close to System z And db link loss budget on all cable types is reduced at 8G What is your current cable distance to the farthest attachment point? Will it have to change? If not already, now is the time to deploy switched-ficon! 8

9 End-to-End FICON/FCP Connectivity Cable distances at 8Gbps 75 th floor 366m of cable run using OM3 MM cable 1,280ft 80-story office tower with a printer on 75 th floor (about 1200 ft from the M/F) Today using 4Gbps you might have need to connect a tape drive or printer at distances up to 380m (1,246 feet) using multimode cable (shortwave) Now you want to deploy 8Gbps soon But 8Gbps will only reach out to 150m (492 feet) so that tape drive or printer might have to be relocated down to the 41 st floor or you might change interfaces and cables to run Single-mode to regain the lost distance 9

10 End-to-End FICON/FCP Connectivity FICON Director cable distances at 8Gbps SM SM MM MM FICON Director OS1 9m using 10km transceivers OS1 9m using 4km transceivers OM3 50m OM2 50m 50 meters 164 feet 150 meters 492 feet LWL(10km) ELWL(40km) 6.2 or 49.7 miles 4 km 2.5 miles Majority of local storage connections are still short wave What is your current cable distance to farthest storage attachment point? 4Gbps allowed it to be 1,246 feet (380m) away using OM3 Will you have to relocate some storage closer in at 8G? Do you need to re-cable from OM2 to OM3 or up to OS1? 10

11 Provide for Greater Distance Connectivity long-wave short-wave Point-to-Point Deployment of FICON 10 KM native Only 40 buffer credits per CHPID Point-to-Point 150 meters native Distance Extension DCX with FX8-24 FCIP Blade Long-wave connections can push a native unrepeated, FICON connection up to 10km This is from mainframe to storage port (LX) And be careful as FICON Express8 has only 40 buffer credits per CHPID In a switched-ficon environment, the switch/director acts like a repeater so that you get the full distance on each side of the switching device 150m to 10km from mainframe to switch device 150m to 25km from switch device to storage When designed with well-thought out placement in mind, a FICON switching device can help alleviate some concerns about distance connectivity even in a local computing environment Locally, can send frames up to 25km (LX) Can use 7800 / FX8-24 Blade for FCIP extension Over 1,300 buffer credits provides long distance (on 1 port of each port blade ASIC) Tape emulation and Global Mirror (XRC) emulation are supported for FICON 11

12 Mainframe Channel Cards FICON Express2 z10, z9, z990, z890 Longwave (LX) to 10km Shortwave (SX) 1 or 2 GBps link rate 270MBps FD max thru-put 68% of 400MBps potential 107 Buffer Credits per port 2G full frame / port 2G half frame / port 2G 819 byte payloads FICON Express4 z10, z9 4km & 10km LX Shortwave (SX) 1, 2 or 4 GBps link rate 520MBps FD max thru-put 65% of 800MBps potential 200 Buffer Credits per port 4G full frame / port 4G half frame / port 4G 819 byte payloads FICON Express4 provides the last native 1Gbps CHPID support FICON Express8 z10 10km LX Shortwave (SX) 2, 4 or 8 GBps link rate 740MBps FD max thru-put 46% of 1600MBps potential 40 Buffer Credits per port 8G full frame / port 8G half frame / port 8G 819 byte payloads FICON switching devices will provide BCs for long distances 12

13 Switched-FICON Provides additional BCs To the left is just an example of the limitations of buffer credits provided on mainframe CHPIDs FICON Express8 40 Buffer Credits per port 8G full frame / port 8G half frame / port 8G 819 byte payloads FICON switching devices can provide as many as 1,300 buffer credits on a single port If a dark fiber connection, or any long distance connection, requires more than 5-10km of distance then switched-ficon can provide connectivity ports that can reach far further, at full path utilization, than a CHPID can provide 13

14 Availability After A Component Failure Point-to-Point Deployment of FICON CHPID path Fails Both sides are down X Cable or optic failure occurs X Avoid complete path failures BUT Storage Port Remains Available! A failure of a FE8 card or FE8 channel port or failure of the P-2-P cable or failure of the storage port optic or storage adapter causes: FE8 port to become unavailable AND Storage port becomes unavailable for everyone! A failure anywhere affects both the mainframe connection and the storage connection Lose an SFP and lose host + storage connect The WORST possible reliability and availability is provided by a P-2-P topology! FC optics are most likely element to fail in the channel path same as in a SAN FC cables are second most likely failure in a fabric In a switched-ficon environment, only a segment is rendered unavailable: The non-failing side remains available If the storage has not failed, its port is still available to be used by other CHIPDs 14

15 Fan In Fan Out For Better Efficiency It is a common practice in a storage network to share one port on the storage subsystem among multiple CHPIDs (HBAs) from multiple servers (partitions). Fan In: the common practice of attaching many underutilized server connected switch ports to a single storage subsystem port as long as additional bandwidth remains and the FC switched fabric is non-blocking Fan Out: the common practice of oversubscribing storage connected switch ports by connecting each of them to many server ports, making the best use of that critical resource. The SAN fan-out ratio of storage ports typically ranges from 4:1 to 12:1 server-to-storage subsystem ports bandwidth dependent. The intent is to fully utilize available storage port bandwidth while enabling the maximum throughput of each HBA (CHPID) to achieve nearwire-rate throughput at a given time. The ratio also implies that the server (partition) ports are being underused (5-50% of possible bandwidth) most of the time. 15

16 FI-FO To Overcome System Bottlenecks System z Fiber Cable FICON Director Cascaded FICON FICON Director DASD /4/8Gbps Per CHPID (transmit and receive) Example Fan In: To one CHPID = 12 (trying to keep the CHPID busy) 380 2Gbps 760 4Gbps Gbps Gbps Per Link (transmit and receive) Example Fan Out: From 12 Storage Adapters Total FICON path usually does not support full speed Must utilize Fan In Fan Out to utilize CHPID port wisely Multiple I/O flows funneled over a single channel path Direct attached FICON is really not a best practice MBps 16

17 DASD Fan In Fan Out Maximize CHPID Capacity Utilization Storage-to-Mainframe 1 : 5 FAN-IN (~510 MBps possible) 460MBps CHPID 110MBps 120MBps 50MBps 80MBps 100 MBps Storage Port Storage-to-Mainframe 1 : 5 FAN-Out (~1800 MBps possible) Maximize Storage Port Capacity Utilization Mainframe-to-Storage 1 : 5 FAN-IN (~380 MBps possible) 360MBps Storage Port 75MBps 100MBps 20MBps 65MBps 100 MBps CHPID Mainframe-to-Storage 1 : 5 FAN-Out (~2550 MBps possible) NOTE: Remember that at 8Gbps a Command Mode z/os CHPID can do about 510MBps maximum 17

18 Fan In Fan Out for Tape Maximize CHPID Capacity Utilization Tape-to-Mainframe 1 : 1 FAN-IN (~510 MBps possible) 320MBps CHPID 320MBps Tape Port Tape-to-Mainframe 1 : 1 FAN-Out (~320 MBps possible) Maximize Tape Port Capacity Utilization Mainframe-to-Tape 1 : 1 FAN-IN (~160 MBps possible) 320MBps Storage Port 320MBps CHPID Mainframe-to-Tape 1 : 1 FAN-Out (~160 MBps possible) NOTE: Remember that at 8Gbps a Command Mode z/os CHPID can do about 510MBps maximum 18

19 Fan In Fan Out On A System Basis Too 12 Paths FAN-IN FICON Directors ESCON had performance constraints that effectively circumvented its ability to provide Fan In-Fan Out 8 Paths FAN-OUT 19

20 Minimize Mainframe Channel Card Costs For Example 16 CHPIDS Point-to-Point Deployment of FICON FICON Point-to-Point Same 16 Storage Ports Can use Fan In Fan Out to minimize CHPID ports required as long as required bandwidth requirements are satisfied For MAX 16 Storage Ports One CHPID per each Storage Port expensive! Use only 8 of the CHPIDs to contain $$ Each storage port in a P-2-P connection requires its very own physical port connection on the mainframe FICON Express8 cards are fairly expensive By model, there is a finite limit to the number of FICON channels available Deploying many P-2-P connections can get pretty expensive A high TCO is usually attributed to P-2-P How do you continue to scale when you run out of either mainframe or storage ports? Buy a new mainframe or storage chassis? In a switched-ficon environment, the Fan In Fan Out ratios solve this problem just like it solves other connectivity problems Director/switch ports are less expensive than mainframe FICON Express8 cards If you run out of FICON CHPIDs then simply continue to Fan-Out to more storage ports Or simply use fewer FICON Express8 channel cards on your Fan-In into storage 20

21 Minimize Mainframe Channel Card Costs For Example 16 CHPIDS Point-to-Point Deployment of FICON FICON Point-to-Point Same 16 Storage Ports Can use Fan In Fan Out to minimize CHPID ports required as long as required bandwidth requirements are satisfied For MAX 16 Storage Ports One CHPID per each Storage Port expensive! Use only 8 of the CHPIDs to contain $$ Each storage port in a P-2-P connection requires its very own physical port connection on the mainframe FICON Express8 cards are fairly expensive By model, there is a finite limit to the number of FICON channels available Deploying many P-2-P connections can get pretty expensive A high TCO is usually attributed to P-2-P How do you continue to scale when you run out of either mainframe or storage ports? Buy a new mainframe or storage chassis? In a switched-ficon environment, the Fan In Fan Out ratios solve this problem just like it solves other connectivity problems Director/switch ports are less expensive than mainframe FICON Express8 cards If you run out of FICON CHPIDs then simply continue to Fan-Out to more storage ports Or simply use fewer FICON Express8 channel cards on your Fan-In into storage 21

22 Robust General Scalability Core DCX DCX 4S 6140 B5300 DCX DCX FICON switching allows for dynamic connectivity in a local or remote environment Point-to-Point does not allow for easy, dynamic growth and scalability One mainframe port is tied to one storage port In a switched-ficon environment, you can provide dynamic connectivity Better use of all channel resources Better use of all storage resources (fan in-fan out) In a switched-ficon environment, you can provide dynamic scalability if you implement FICON cascading Better use of all channel resources Better use of all storage resources Fan in-fan out Efficient utilization of all resources Quick response to new connectivity demands Proven Core-to-Edge connectivity 22

23 Scalability Beyond System z CHPID Limits Mainframes Have a limited Number of FICON CHPIDs Attach to many more devices than possible with P-2-P z800: 32 FICON Express z900: 96 FICON Express z890: 40 FICON Express z890: 80 FICON Express2 z990: 120 FICON Express2 z990: 240 FICON Express2 z9bc: 112 FICON Express4 z9ec: 336 FICON Express4 z10bc: 112 FICON Express8 z10ec: 336 FICON Express8 ICL d DCX Maximums Can use DCX ICL s and FICON Cascading to act as a CHPID multiplier for obtaining access to storage devices Each storage port in a P-2-P connection requires its own physical port connection on the mainframe There is a finite limit to the number of FICON channels depends upon the model of the mainframe that you are using What happens when you need more FICON connectivity than you have CHPIDs? How do you continue to scale when you run out of mainframe CHPIDs? Even if you make really good use of of Fan- In, this could eventually happen In a switched-ficon environment, the Fan In Fan Out ratios solve this problem just like it solves many other connectivity and scalability problems If you run out of FICON CHPIDs then simply continue to Fan-Out to more storage ports Or simply use fewer FICON channel cards on your Fan-In into storage depending upon your bandwidth requirements 23

24 Balance Workload Across All Storage Ports Point-to-Point Deployment of FICON CHPIDs 8 Path Group FICON Point-to-Point Path Busy % 20% 20% 20% 20% Overall low utilization but no way to balance workload. Use only 8 of the CHPIDs to contain $$ FICON DCX Can use Fan In Fan Out to help distribute workload across all of the storage ports evenly while making best use of high-capacity storage arrays Maybe only buy 10 of 16 Storage Ports IF CHPIDs 10 through 17 (8 path group) are consistently pushing low amounts of data, there is really no opportunity to make better use of either those channel ports or those storage ports FICON does not do channel disconnect With P-2-P no sharing of ports is possible Capability of storage device may be under utilized as a result of these P-2-P connections In a switched-ficon environment, Fan In to Fan Out ratios help evenly distribute storage workload across all ports Fewer channel ports can often actually push more bandwidth across fewer storage ports Can typically put more capacity inside of a DASD array when using Fan In Fan Out rather than P-2-P while using equal or fewer total storage connections 24

25 Mainframe and Linux Resource Sharing Point-to-Point Deployment of FICON FICON FCP Will need to provide unique CHPIDs and maybe storage ports, some for Linux and some for FICON and possibly even different storage arrays FICON FCP FICON and Linux Intermix Point-to-Point Point-to-Point Many customers are adopting Linux on System z Use mainframe functionality to manage up to several thousands of Linux clients running on your System z Use Node Point ID Virtualization (NPIV) to interleave I/O across physical channels IFL engines keep software costs down In a switched-ficon environment, the switch/director accepts both FICON and Linux FC connections and can then maximize the use of storage ports via reasonable Fan In Fan Out ratios Better scalability and access to storage ports Lower TCO for both FICON and Linux Easier manageability of both FICON and Linux Might want to have FICON using some storage arrays and Linux other storage arrays Open systems maintenance might be on a different schedule than FICON maintenance 25

26 A Simplified Schematic Linux on a System z without NPIV NPIV is ONLY available in a switch-ficon fabric! Linux on System z without NPIV Linux Guests Linux A V M A I/O A One FCP CHPID per Linux guest A A A M-Series and B-Series Linux B B B B B B Linux C Linux n C n IOS Linux on System z can run in its own LPAR(s) but usually it is deployed as guests under VM C C C C n n n n No parallelism so it is very difficult to drive I/O for lots of Linux images with only 256 CHPIDs Probably very little I/O bandwidth utilization per CHPID and switch port 26

27 Linux on System z using NPIV System z using NPIV Linux Guests Linux A Linux B Linux C Linux n V M A B C n I/O IOS NPIV works only when using switched-ficon One FCP channel for many Linux guests n C B A Lots of Parallelism Fewer switch ports required! 8 Gbps Is Great For NPIV! M-Series and B-Series NPIV enabled Much better I/O bandwidth utilization per path 27

28 How Are Directors and Switches Different B5100 B5300 M-Series can run at up to 400MBps and B-Series can run at up to 800MBps on a port by port basis M6140 Director Mi10K Director DCX-4S DCX SAN Switches Good Availability up to 99.99% Based upon motherboard design Some redundant components like power supplies and fans Fiber Channel ports Good fabric Scalability (100 s of ports) Online microcode activation Online Health Monitoring Online Error Detection Online Fault isolation checking It is not when it is working, but rather when a problem occurs, that truly differentiates a Director from a Switch! SAN Directors Superb Availability up to % Based on discrete, redundant parts Redundancy and hot swap FRUs throughout the architecture Highest port counts up to 384 ports Superior fabric Scalability (1,000s of ports) Online microcode activation Online Health Monitoring Online Error Detection Online Fault isolation checking Online Error Recovery (non-disruptive failover) Online Repair of the error (hot swap) 28

29 How Are Directors and Switches Different Since switches are motherboard-based, they are engineered to run at the then current line rate Today each port of a B5100 and B5300 can run at 8Gbps Failing SFPs can be hot-swapped but physical ports cannot be replaced A switch must be completely replaced to repair a failed physical port(s) Directors have discrete, redundant components that are engineered to run at the then current line rate Today each port of a DCX and DCX-4S can run at 8Gbps Failing SPFs can be hot-swap replaced (and fans and power supplies ) New blades can replace blades that have failing or failed physical ports Brocade expects to have 16Gbps blades for the DCX and DCX-4S within the next 12 months The next generation mainframe will be engineered for 16G CHPIDs A DCX and DCX-4S should be upgradable in the future to 16Gbps non-disruptively (swap out old 8G blades swap in new 16G blades) But 8G switches will have to be completely swapped out and replaced with 16G capable switches to achieve 16G fabrics 29

30 The DCX/DCX-4S is a GREAT Chassis for FICON! Highlights The DCX/DCX-4S has an internal cycle time of ~1 microsec DCX/DCX-4S has 32 times more chassis bandwidth than previous, popular Brocade FICON Directors DCX/DCX-4S scales up to as many as 2,304 ports per fabric through use of our ingenious, unique inter-chassis links DCX/DCX-4S was built for 16G link speeds while currently shipping with only 8G capable blades and SFPs DCX/DCX-4S has the best energy efficiency, for FICON Directors, in the world! 30

31 The DCX/DCX-4S is a GREAT Chassis for FICON! Technical Description Overview 512Gb aggregate bandwidth per slot (256Gb transmit & receive) 16 ports/blade = 256Gb / 16p = 16.0Gb/port potential (gated by 8G SFPs) 32 ports/blade = 256Gb / 32p = 8.0Gb/port potential (1:1) 48 ports/blade = 256Gb / 48p = 5.3Gb/port (get to 1:1 via Local Switching) A full, real 2Tb of bandwidth for internal frame handling DCX; DCX-4S has 1Tb of bandwidth A full, real 3Tb of bandwidth when local switching used DCX; DCX-4S has 1.5Tb of bandwidth Chassis-to-Chassis bandwidth of.5tb when using ICLs DCX: ICL s allow FICON scaling of up to 2,304 ports per fabric! DCX-4S: ICL s allow FICON scaling of up to 1,152 ports per fabric! The DCX can pass up to BILLION frames per second 31

32 Just Say NO to Direct-Attached FICON! Point-to-Point Deployment of FICON FICON Point-to-Point As a Best Practice, never direct attach FICON! 32

33 THE END I hope this information was useful to you!