Advanced FICON Networks. Stephen Guendert BROCADE February 15, 2007 Session 3809

Transcription

1 Advanced Networks Stephen Guendert BROCADE February 15, 2007 Session 3809

2 Today s Agenda Data Center I/O Technologies Advanced Fabric Services and FC Intermix Cascading Cascaded Link Management Director Activity Report 2

3 Normal SAN Data Center I/O Technologies S C S I SAN FC 3

4 Data Center I/O Technologies Intermixed Normal SAN and / Separate Intermixed Network w/ on the Edge SCSI/MVS S C S I z9 zos Linux VM DELL IBM SAN I-M I-M SAN EMC On The Edge SUN HDS Shared Storage SUN HP SUN FC ESCON CTC ISL4

5 Data Center I/O Technologies All Infrastructure CTC for and Sysplex and Bridge GDPS Card MVS SCSI/MVS S C S I z9 Z8xx / z9xx zos TPF zos Linux VM DELL IBM SAN Channel-to-Channel Bridge Card E S C O N I-M I-M SAN ESCON IBM On The Edge EMC SUN HDS SUN HP SUN FC ESCON CTC ISL5

6 Data Center I/O Technologies MVS SCSI/MVS S C S I z9 Z8xx / z9xx zos TPF zos Linux VM DELL IBM Channel-to-Channel SAN Bridge Card E S C O N I-M I-M SAN ESCON 1 Hop ISL IBM 1 Hop ISL Cascaded Link UltraNet EMC On The Edge SUN HDS Shared Storage SONET / ATM / FCIP SUN Dark Fibre UltraNet HP IFC ESCON ISL SUN IBM EMC HDS SUN EMC HDS 1 35 km away Data Center Long Distance Data Center FC ESCON CTC ISL6

7 Why Worry About Long Distance BC/DR? Terrorist Attack on New York City, 9/11/01 Many companies in Trade Towers went out of business Power Blackout, Northeast USA and Canada 8/14/03 9,300 sq. miles, millions of people and thousands of companies Power Blackout, Italy and France 9/28/03 All of Italy except the island of Sardinia and some of France Southeast Asia Earthquake 12/26/04 Affected Indonesia, Thailand, India, Sri-Lanka, Malaysia, Myanmar, Bangladesh, Maldives, Reunion Island (French), Seychelles, Madagascar, Mauritius, Somalia, Tanzania, Kenya, Oman, South Africa and Australia Hurricane Katrina, Southeast USA 8/29/05 Affected 90,000 square miles (more than two large states) 7

8 ESCON Conversion

9 to Legacy ESCON Converter Converters provide native connectivity to legacy ESCON Devices while offering: Investment protection for ESCON Devices you already own Reduced capital and operating costs associated with migration Host and storage consolidation, infrastructure simplification and an ability to take advantage of the many benefits Not an ISL! Supports multiple CPU access Tape DASD Converter ESCON 9

10 Originally Supported Configurations Local Connectivity Direct Attached ESCON Direct Connect Mainframe to ESCON Device Conversion ESCON Single Director Fabric to ESCON Device Conversion MAN Connectivity ESCON Cascaded Directors Cascaded Switches to ESCON Device Conversion MAN connectivity 10

11 Intermix

12 /FC Intermix Basic Topology and devices mainframe Linux on mainframe servers / storage A Mainframe Only Intermix Environment Linux Linux Fibre Channel Management -centric Zoning utilized -centric Zoning utilized HCD/HCM definitions CUP support Intermix with a SAN 12

13 Protocol Intermix Most Enterprises Are Not Considering Going to Enterprise Intermix WHY? Control! Mainframe and Open Systems are often different departments and Open Systems may use different budgets Maybe more importantly, vendors usually discuss switch interoperability and intermix as a tactic Should You Consider Enterprise Intermix? YES - but maybe it s better as a Strategic Initiative not as a tactic! 13

14 Protocol Intermix in an Enterprise Mainframe and SAN are generally independent complexes IBM z990 zos TPF Direct Attached ESCON IBM Tape SUN Tape Direct Attached ESCON IBM z800 Isolated Mainframe System Isolated SAN HDS Storage Area Network Dell HDS HP SUN EMC EMC 14

15 Protocol Intermix For Storage Consolidation The Major Considerations Are Now Management and Control! Maybe Upgrade to a z9 Maybe Linux IBM z9 zos TPF Linux? plus FC Links Tape Tape Links IBM z800 Dell IBM HDS HDS Capable Storage Of Choice Utilize mod-9 or mod-27 volume sizes and still get GREAT performance System Area Network HP SUN Capable Storage Of Choice Far fewer boxes to manage and vendor tools that you must utilize IBM EMC 8830 EMC

16 CUP Zoning (FCZ) Extends CUP implementation Provides secure management when host(s) use CUP in an intermix environment FCZ - Mainframe Open system devices in a SAN zone protected against inadvertent changes in configuration by any of the hosts Prevents mainframe host using CUP from blocking or unblocking any ports which are included in an Open Systems SAN partition FCZ - FC SAN Open system host(s) cannot manipulate the state of mainframe devices in FCZ as they do not know exist side of the intermixed environment is protected 16

17 CUP Zoning Below Is A Typical Intermix Infrastructure FCZ for Linux Partition Mainframe Linux Linux - FCZ for Partition CUP Enabled M/F DASD Without FCZ, CUP Sees All Manages All Mainframe host queries to switch are filtered through active zoning configuration Provides port isolation capabilities across switching complex Active zoning configuration applied to FMS CUP feature, creates a set of partitions for the environment Traditional zones become CUP Partitions and isolate CUP management capabilities to ports Open Systems FCZ for SAN Partition(s) Open DASD Consider deploying in any -SAN intermixed environment 17

18 Cascading

19 Cascading WITHOUT CASCADING Typical topology Redundant cabling Devices connected to multiple Directors for sharing More paths between the mainframe servers and the Directors = more CHPids Higher cost of ownership More difficult to manage S i t e 1 S i t e 2 z-series Tape & Tape Libraries Tape & Tape Libraries 19

20 Cascading WITH CASCADING Efficient cross-site connectivity S i t e z-series Tape & Tape Libraries Fewer CHPid ports 1 Less fiber cabling Easier to manage More scalable Lower cost of ownership Highly Available Redundant Fabrics S i t e 2 Tape & Tape Libraries 20

21 Cascading Use ISLs for mainframe environments Increased flexibility Fabric cost savings Optimization of storage resources Improved disaster recovery and business continuance 1.0 Gbps 100 MBps (now) 2.0 Gbps 200 MBps (now) 4.0 Gbps 400 MBps 10Gb (6.5 Gbps) 740 MBps 10.0 Gbps 1200 MBps Inter-Switch Links ISLs Cascaded Links 21

22 Cascading Meshed Topology IBM certifies up to 7 Directors and/or switches in a mainframe fabric per CHIPID Standalone Switching zseries Processor Cascaded Switching This Path Would Not Be Supported!. Only homogeneous cascaded configurations are allowed Control unit must be only one hop from the host Multi-hop probably won t happen in 2007 Scalable Growth Use a Large director as hub Mix of directors and switches at the edge All of the Directors/Switches in this fabric must 22 come from the same vendor must be homogeneous.

23 Be Careful When Using Cascaded Express2 Cards Linux Linux Linux So Here is how we configured our system for cascading 5 Channel Cards 26MBps 10MBps 16MBps 20MBps 22MBps 21MBps 12MBps 20MBps 10MBps 18MBps 18MBps 13MBps =< 75% =< 75% 140MBps 140MBps =< 75% =< 75% 140MBps 140MBps 56 Director Ports (44 for F-ports) (9 for E-ports supporting 1,366MBps) (3 spares) 200MBps ports 8 Tape Drives Nine 2Gbps ISLs 200MBps ports 16 Storage Ports Storage Array Cache and Processors 23

24 Be Careful When Using Cascaded Express2 Cards Linux Linux Linux 5 Channel Cards 26MBps 10MBps 16MBps 20MBps 22MBps 21MBps 12MBps 20MBps 10MBps 18MBps 18MBps 13MBps =< 75% =< 75% 140MBps 140MBps =< 75% =< 75% 140MBps 140MBps FSPF allocated the ISLs for connectivity but not for performance. For example, you can see that all of the tape (560MBps) was given only 3 ISLs but it needed 4. A similar problem exists for the DASD connectivity shown as. For those links, only a single ISL was allocated. 200MBps ports 200MBps ports 16 Storage Ports Storage Array Cache and Processors 24

25 Why 2-Hop Would Be Nice For native TAPE, having a capability of doing 2-hop would be financially beneficial S i t e 1 z-series 2 HOPs Local Tape 1 HOP 1 HOP Not needed for a Point-to-Point Virtual Tape system S i t e 2 2 HOPs Remote Tape 25

26 Cascaded Link Management Mainframe folks like to control everything! The word Control Freak comes to mind ISL links not specified in HCD ISLs cannot be directly controlled in fabric infrastructure Not in HCD/IOCD Can t be measured with traditional MF tools Vanilla RMF collects no performance statistics on ISL links Must find a way Control how cascaded links are distributed and utilized Measure the impact of cascaded links in our environment 26

27 Default ISL Allocation FSPF Fabric Shortest Path First (FSPF) Devices distributed across links based on link cost Uses Round Robin technique Static routing unless there is a fabric change No direct mechanism for selecting routes 27

28 Prohibit Path (Allow/Prohibit Matrix) Tape 01 Tape- 02 Tape- 03 Tape- 04 Right click actions Prohibit Row Allow Row Prohibit All Allow All Block All Unblock All Clear All Can Save Several in Configuration Library Unlike ESCON, a port can talk to itself, which is very useful for CTC Hardware Enforced Port Blocking (PDCMs) 28

29 ISL Allocation using Prohibit Paths Block some ISL ports Configure prohibited ISL paths Enables us to influence ISL route selection Must be done for each and every switch device Time consuming and very static in nature Uses negative logic: ISLs that cannot be used to connect X X X X XX X X Must configure both sides No failover from a cascaded link failure 29

30 ISL Allocation using Preferred Path YOU choose the ISL route! Enables influence of cascaded link selection Defined for F-port to E-port ISL failure results in normal failover behavior Uses Positive logic is used: ISLs that can be used to connect Preferred Preferred Must be done for each and every switch device 30

31 Combining ISL Allocation Methodologies Can combine Prohibit Path & Preferred Path Preferred Prohibit Should configure both sides + 31

32 Open Trunking Open Trunking provides automatic response to the over-subscription of ISL link Enabled on a switch-by-switch basis License Key Pay to use option OpenTrunking tm PFE OpenTrunking tm PFE Over-subscription threshold is per ISL link It can be configured! 32

33 10Gbps - A Two-edged Sword No auto-negotiation with 1/2/4Gb devices Uses 64b/66b encoding scheme, not FC industry standard of 8b/10b used for 1/2/4Gb ANSI committee decision Auto negotiation requires the same encoding at both ends 10Gbps is used only for Cascaded Links (ISLs) 64b/66b is more efficient for moving data in the FC frame 10Gbps developed for SONET with objective of minimizing overhead 8b/10b encoding one byte of check data per four bytes of user data 64b66b encoding One byte of check data per thirty-two two bytes of user data 10Gbps Cascaded Link (ISL) - more efficient data transfer 33

34 Cannot Auto-Negotiate! Why Use 10Gbps? 10Gbps - uses 64b66b Data Frame Encoding 64b66b encoding causes 3% overhead in the data frame -- 2 of every 66 bits is overhead 64 bits of data bits of data ,400,000 8b10b - The same as one 2,112 byte payload for check data out of every 40 frames 64b66b - The same as one 2,112 byte payload for check data out of every 265 frames additional data frame payloads per day per 10Gbps 600MBps! 1, 2, 4Gbps - uses 8b10b Data Frame Encoding 8b10b - At 600MBps, same as 7,200 out of 297,900 frames for check data 64b66b - At 600MBps, same as 1,200 out of 297,900 frames for check data DATA FRAME 8b10b encoding causes 20% overhead in the data frame -- 2 of every 10 bits is overhead 34

35 Comparing 4Gbps to 10Gbps for ISLs 6xxx i10k ----4Gbps Gbps Typically 4Gbps from all vendors is supported only on oversubscribed line cards As an 8b10b technology, architect to <=75% utilization 400MBps raw bandwidth should run at <= 300MBps Small frame payloads will cause low bandwidth (B/W) movement but high link utilization so be careful of that Two 4Gbps ISL links provide no more than 600MBps B/W Care must be taken to manage the oversubscription of 4Gbps line cards if used for cascaded links (ISLs) Gbps Gbps Unlike 4Gps, 10Gbps ports are not oversubscribed As a 64b66b technology, architect to <=80% utilization 1,200MBps raw bandwidth should run at <= 960MBps Small frame payloads will cause low bandwidth movement but high link utilization so be careful of that Two 10Gbps ISL links can provide about 1,920MBps bandwidth Would take more than six 4Gbps links to equal two 10Gbps links Will probably not become much cheaper over time 35

36 Best Practices for Deploying and Managing Cascaded Links These are the best four options to deploy since they are dynamic and are not labor intensive! FSPF Automatic - FC standards default Choose least-cost ISL and then round-robin at login Automatic function that requires no manual configuration or human effort Open Trunking tm Automatic de-congestion! An automatic means to dynamically detect and correct over-congested links Let FSPF do initial Cascaded Link allocation then OT will keep them un-congested Both are automatic, dynamic functions requiring no manual configuration 4Gbps Provides a nice sized cascaded link for getting data frames across a network quickly 10Gbps port card Provides the most effective cascaded link for getting data frames across a network Preferred Path Manually configure F-ports to specific E-ports to Influence FSPF Route Selection Static but there is failover to another cascaded link from a failed cascaded link Prohibit Path Prohibit Dynamic Connectivity Matrix Done at the Hardware Routing Table blocks ports from accepting data frames Manually Allow/Prohibit cascaded links to influence FSPF allocation algorithms Static and there is no failover from blocked cascaded links 36

37 Controlling Cascaded Links - Example Customer Requirement: zos, TPF and Linux are all running on a z9 Isolate the ISL usage between these systems Provide Dynamic management of ISL links 4 ISLs per each of the three environments Deployment: Use Prohibit Paths to create logical trunks Create three separate ISL flow groups! Block zos ports from 8 of the ISLs Block TPF ports from 8 of the ISLs Block Linux ports from 8 of the ISLs Use Open Trunking Create separate Trunk groups with Prohibit Paths (zos, TPF, Linux) Open Trunking now manages all groups Independent management No mixing of the frames outside the designated, unique trunk group zos TPF Linux ISOLATED FLOWs zos TPF FC 12 ISLs Linux 37

38 Director Measurements F-port to ISL Interconnection Frame Pacing Delay Avg R/W Frame Size R/W MBps Error Count ISL RMF s Switching Device Report Components Host-to-Director Director-to-Storage 2 / 1 aggregation 2 / 1 aggregation 5 / 2 aggregation 9 Channels FAN-IN 2 / 4 / 10 Gbps ISL s 4 Channels FAN-OUT RMF 74 subtype 7 records turned on and CUP code implemented! 38

39 Questions?