Leverage the Future Today

Transcription

1 Session Q03 Leverage the Future Today A look at today's host to storage performance, tricks and tips for better performance management of your FICON environment, and applying it to the FICON infrastructure of tomorrow Steve Guendert, Principal Engineer sguender@brocade.com IBM System z Expo September 17-21, San Antonio, TX IBM Corporation IBM System z Expo

2 Legal Disclaimer All or some of the products detailed in this presentation may still be under development and certain specifications, including but not limited to, release dates, prices, and product features, may change. The products may not function as intended and a production version of the products may never be released. Even if a production version is released, it may be materially different from the pre-release version discussed in this presentation. NOTHING IN THIS PRESENTATION SHALL BE DEEMED TO CREATE A WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT OF THIRD PARTY RIGHTS WITH RESPECT TO ANY PRODUCTS AND SERVICES REFERENCED HEREIN. Brocade, the Brocade B-weave logo, McDATA, Fabric OS, File Lifecycle Manager, MyView, Secure Fabric OS, SilkWorm, and StorageX are registered trademarks and the Brocade B- wing symbol and Tapestry are trademarks of Brocade Communications or its subsidiaries, in the United States and/or in other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners. 2

3 Abstract A performance optimized FICON environment will result in lower TCO. This session will focus on maximizing the performance of your FICON environment today, so as to minimize your TCO, in preparation for the future enhancements to the FICON protocol. The presentation will be based on several recent zjournal articles written by the speaker. We'll discuss BB credit optimization, QoS, Modified Indirect Data Address Word (MIDAW), what is 4 Gbps really doing for you?, frame pacing delay, Open Exchange management and using CUP and the RMF 74-7 record (FICON director Activity report), as well as other SW for optimizing performance. We'll look at preparing for 8 Gbps FICON: mechanisms for real time counting of BB credits, as well as dynamic allocation of BB credits on an I/O basis. 3

4 Agenda Buffer to Buffer Credit Optimization Quality of Service (QoS) MIDAW What is 4 Gbps FICON doing for me? Open Exchanges Frame Pacing Delay and other RMF 74-7 tidbits Software SW demo-keith 4

5 Intros-Who is Steve Guendert Brocade Principal Engineer, focused on mainframe SHARE Board of Directors (Director of IT) Nominee for Computer Measurement Group (CMG) BoD CMG Storage Subject Chair Academic PhD Coursework completed (CS/MIS) (stats-performance) M.S. in MIS dissertation topic on Enterprise I/O subsystems and designing continuous availability data centers Industry experience IBM, McDATA, CNT, Brocade Ohio Valley CMG Regional Director CMG Editorial Review Board (ERB), zjournal ERB Published papers in zjournal, CMG, NaSPA Technical Support, DRJ 5

6 Buffer to Buffer Credit Optimization Basics on flow control and buffer to buffer credits are at the end of the slides, in the extra material section

7 Buffer Credits relative to Link Speed For full 2112 byte data frames 1 Gbit or 100 MB 2 Gbit or 200 MB 4 Gbit or 400 MB 1 Buffer Credit (BC) 1 Buffer Credit (BC) 1 st bit on the wire to last bit on the wire, the 1 st bit went 2 km.5 full credit/km Takes more data to keep pipe at 100% utilization. That is why customers went from 20% utilization to 10% 2Gbps 1 full credit/km 2 full credits/km 0 km 1km 2 km 3 km 4 km Every time the data rate doubles, you half the maximum distance It is important to remember that block level z/os transfers rarely employ full frames! What happens if you go too far? 7

8 What is the optimal number of BB Credits? Optimal number of credits is determined by: Distance (frame delivery time) Processing time at receiving port Link signaling rate Size of frames being transmitted Optimal # BB_Credit = (Round-trip receiving time+receiving_port processing time) Frame Transmission time 8

9 Optimal number of BB_Credits(2) So, the optimal number of BB Credits depends on 3 key parameters: 1. Round trip time (distance) 2. Frame processing time 3. Frame transmission time* * As the link speed increases, the frame transmission time is reduced; therefore, as we get faster iterations of FICON such as FICON Express4 and Express8, the amount of credits need to be increased to obtain full link utilization, even in a short distance environment! 9

10 Why an optimal number? Analogous to DASD and cache sizing Law of diminishing marginal returns Exceeding the optimal number of BB_credits does nothing to increase performance, it merely increases your costs. Optimal number of BB Credits allows for performance optimized distance solutions. Optimal number allows for better capacity planning How does your director allocate BB Credits? 10

11 Data Droop for Over 2Gb/s Distance versus Buffer Credits Cronin, Performance Considerations for Cascaded FICON Directors, IBM, March 2003 Hence, serious consideration must be give to the assignment of credits to ports on director architectures that share a pool of credits among the ports on a card. While relatively few credits (16) might be assigned to local devices, the bulk of the credits should be assigned to ISLs For data chaining OLTP workloads, assume a worst case 512 byte average credit size to avoid any potential of droop MIDAWs and RTD/WTD substantially increase the average credit size 11

12 What to do? Mechanism for counting BB credit usage currently does not exist Data is there, how do we exploit it? You find out you run out after the fact: let s be more proactive rather than being satisfied with being blind What we wind up with is BB credit assignment overkill, similar to how we overkill PAV aliases. Just as you can run out of addresses, you can run out of BB credits (or will assign them poorly). Do some analysis and plan things better! Save TCO $$ in the process! BB Counting mechanism Dynamic allocation of BB credits (based on HyperPAVs) 12

13 Quality of Service (QoS) Not all data is created equal, nor does all data require equal treatment.

14 Definition of QoS Service: the expected behavior or outcome from a system. QoS: the degree to which the expected outcome is realized. Class of Service (CoS) Defines the type of service but does not indicate how well the service is performed. FC CoS defines message delivery guarantees which is far different from guaranteeing response times, bandwidths, etc. 14

15 SNIA s Definition QoS is a technique for managing computer system resources such as bandwidth, by specifying user visible parameters such as message delivery time. Policy rules are used to describe the operation of network elements to make these guarantees. In a nutshell-in a storage network, QoS is a set of metrics that predict the behavior, reliability, speed and latency of a path. 15

16 So, what s the problem? QoS in storage and storage networks is an optimization problem Achieve optimization by trading one performance metric for another (trade throughput for response time). In general, storage subsystems cannot make QoS guarantees. They are constructed to accept, and queue all arriving I/O commands. Certain performance tuning techniques help achieve partial QoS Queue prioritization, assigning more buffers/cache, DASD spindles etc to high priority jobs Partial QoS is merely a best effort service level that cannot give QoS guarantees. Most existing implementations are partial QoS, that only address performance monitoring and configuration issues in their own little segment of the data center sysplex. These partial QoS implementations are not true end to end QoS, which is what the mainframe environment really requires! 16

17 Some ways FICON QoS is being addressed The failed experiment with Fibre Channel Class 4 CoS Infiniband s Virtual Lanes Virtual Channels Director based software driven mechanisms 17

18 Some ways FICON QoS is being addressed The failed experiment with Fibre Channel Class 4 CoS Infiniband s Virtual Lanes Virtual Channels Director based software driven mechanisms All are partial QoS mechanisms that only drive QoS for the director, not end to end from host to CU! 18

19 End to End QoS Workload Manager Intelligent Resource Director Dynamic Channel Path Management Channel Subsystem Priority I/O queuing 19

20 What is 4 Gbps FICON really doing for me?

21 FICON Channel Implementations Server Channel Type Microprocessor G5/G6 FICON 166 MHz Power PC 750 Internal Bus To STI 32-bit 33MHz Data Rate STI 1 Gbit 1 z900 FICON 333 MHz Power PC 750 L2 Cache z8xx/z9xx z890/z990 z9 z9 FICON Express FICON Express 2 FICON Express MHz Power PC 750 L2 Cache 448 MHz Power PC 750 L2 Cache 448 MHz Power PC 750 L2 Cache 32-bit 33 MHz 64-bit 66 MHz 64-bit 112 MHz 64-bit 112 MHz 1 Gbit 1 2 Gbit 1 2 Gbit 1 2 Gbit 2 21

22 FICON Channel Full Duplex Read/Write Mix Data Transfers Reference: IBM 22

23 FICON Channel Performance-Start I/Os Reference: IBM 23

24 What to do? Are you driving your 2 Gbps channels hard enough to benefit? Better bandwidth, but not increased I/O Are you running MIDAW? If not, you ll probably see little benefit to running 4 Gbps? Have I modeled my environment with 4 Gbps to see if I will gain anything? Are the gains worth the cost? How much will it cost me to implement 4 Gbps FICON? Should I wait for the new 4 GbpS channel cards with better I/O capability? What about my storage? 24

25 MIDAWs Modified Indirect Data Address Words

26 Intro MIDAW introduced with z9 to address: Limitations in the existing IDAW implementation Inefficiencies with FICON data chaining FICON channels historically have experienced persistent limitations for achievable data rates for small block size traffic FICON upper level protocol requires that information units (IUs) contain a single CCW, a CCW and its associated data, or only data. Achieves far lower I/O rates for small blocks than open systems in exchange for better data integrity and recoverability. Small blocks require more CCWs per track than large blocks- channels are less efficient for small blocks. 26

27 Channel Efficiency Small block operations exhaust the microprocessor: channel busy metric Large data transfer operations saturate the bus: Bus busy metric Channel efficiency: ratio of bus busy to channel busy, is an indication of the effective data transfer rate at saturation. Effective employment of the FICON channel bandwidth requires balancing channel and bus busy Reference: Dr. Pat Artis Understanding the Performance Implications of MIDAWS,

28 What is the MIDAW Facility? It is a system architecture and software exploitation New method of gathering and scattering data into and from noncontiguous System z9 storage locations during I/O operations By reducing the number of CCWs in some channel programs, FICON performance can be improved in certain applications such as DB2 28

29 Improvement in FICON Express 4 Channel Processor Utilization and Bandwidth Reference: Cathy Cronin IBM System z9 and FICON Express 4 Performance Update. SHARE Tampa Proceedings, Feb

30 More bytes/frame more efficient channel Reference: Cathy Cronin IBM System z9 and FICON Express 4 Performance Update. SHARE Tampa Proceedings, Feb

31 FICON Express 4 Experiment Reference: Dr. Pat Artis Understanding the Performance Implications of MIDAWS,

32 MIDAW Conclusions MIDAW improves both the channel efficiency and service times for FICON Express 4 and FICON Express 2. MIDAW dramatically increases the maximum SSCH rate achievable by subsystems. MIDAW increases the efficiency of your channel links by transferring more data frames and less command frames. Today, use of MIDAW is really the only way to get a benefit from moving to FICON Express 4. 32

33 Open Exchanges

34 Open Exchanges An Open Exchange (OE) is a logical resource that represents an I/O being processed by a channel. An OE number is assigned to each SSCH request. FICON and FICON Express/EX2 channels are limited to 32 concurrent OEs for z8x0 and z9x0 processors. FICON Express 2 and FEX4 on the z9 is limited to 64 concurrent OEs. 34

35 Open Exchanges (2) An OE is a conversation number that is assigned by the fibre channel port. It is a unique hardware generated number that is assigned to every work unit related to a specific I/O. This value is contained within the frame header. Each work unit is also assigned a sequence number by the fibre channel port. The number of OEs for a channel can be estimated using Little s Law. 35

36 Subsystem Interface Open Exchanges ANALFIOE adds a per device OE value to the MXG Type 74 file. Current DASD subsystem FICON interfaces are capable of supporting at least 48 concurrent OEs Manage them to values less than 20 36

37 Why are Open Exchanges important Artis study demonstrated a lack of correlation between increasing service times, and channel utilization/bus utilization. Correlation was found to exist between increasing service times and open exchange saturation Concept of open exchange steam gauge for managing FICON channels. 37

38 Estimating the number of active OEs The number of Open Exchanges is NOT reported by RMF. A calculation/estimation must be done to obtain this number Active_OEs = Chan_I/O_Rate * OE_Duration For the OE_Duration, we now use the sum of the average CMR, DISC, and CONN times for the I/Os performed by the channel Active_OEs = Channel_I/O_Rate * (CMR + DISC + CONN) MXG member ANALFIOE performs this calculation Intellimagic s RMF Magic SW performs this and graphs the trends CMR=Command Response Time 38

39 Frame Pacing Delay and other RMF 74-7 tidbits The FICON Director Activity Report is your friend

40 Frame Pacing Delay AVG FRAME PACING Defined by RMF as the average time (microseconds) that a frame has to wait before it could be transmitted due to no buffer credits being available. You always want to see a zero value in this field! Reporting on this value was one of the primary reason that the RMF 74-7 record was developed it was not needed for ESCON A non-zero value in the AVG FRAME PACING field indicates that you have an issue with insufficient BB Credits It is critical to use CUP in any FICON environment in which distance extension is being utilized 4Gbps may create more Frame Pacing Delay issues than 2Gbps z/os disk workloads rarely use a "full" 2148 byte credit For example, with a 4k block transfer, the average frame size for each 4k transfer is typically about 819 bytes 40

41 FICON Director Measurements F-port to ISL Interconnection Frame Pacing Delay Avg R/W Frame Size R/W MBps Error Count ISL RMF s FICON Switching Device Report Components Host-to-Director FICON Director-to-Storage FICON 2 / 1 aggregation 2 / 1 aggregation 5 / 2 aggregation 9 Channels FAN-IN 2 / 4 / 10 Gbps ISL s 4 Channels FAN-OUT RMF 74 subtype 7 records turned on and CUP code implemented! 41

42 FICON Director Activity Report AVG FRAME PACING: the average time (in microseconds) that a frame had to wait before being transmitted due to no buffer credits being available AVG FRAME SIZE READ/WRITE: the average frame size (in bytes) used to transmit and receive data during this interval PORT BANDWIDTH READ/WRITE: The rate (in MB/sec) of data transmitted and received during this interval ERROR COUNT: The number of errors counted during this interval 42

43 Frame Pacing Delay Being Reported 43

44 Local Frame Pacing Delay How can you run out of buffer credits inside a datacenter? Frame pacing delays occur when multiple, heavily used paths merge into a single FICON link Frame pacing delays can contribute to PEND, DISC, and CONN time measurements Frame Pacing Delay is caused by running out of buffer credits! Frames Frames 100 MBps Frames 300 MBps 100 MBps Frames 100 MBps Now this CONGESTED cascaded link is causing additional PEND and/or CONN time to many storage ports and possibly additional IOS Queue Time to some of the CHPIDs UCBs serviced by these storage ports are probably experiencing additional delays usually reported as PEND Time and CONN Time and sometimes as DISC time Frame Pacing Delay came about with FC and FICON so it is not a factor in ESCON performance! 44

45 Long Distance Frame Pacing Delay An example using a 20km distance Frame Pacing Delay on M6140 Frame Pacing Delay on zos disk workloads very rarely produce full fibre channel frames (4K transfers have an arithmetic mean of 819 bytes for frame size) 45

46 Software that can help Brocade EFCM Intellimagic RMF Magic Brocade SANHealth FICON

47 Brocade EFCM Brocade EFCM provides sophisticated management capabilities for Brocade fabrics Visualization/Discovery Monitoring and reporting Real-time and historical data collection Zoning and Zone Library Management Diagnostic tools/fault isolation Security features Advanced Call Home Highly scalable architecture Optional advanced features 47

48 SAN Health TM for FICON Discovery, documentation and Planning tool Works on all McDATA & Brocade products Offers migration and conversion tools SAN Health Professional Optional Change analysis Provides inventory management and search engines FICON Features Include IOCP files in analysis Display sequence number in reports and graphics Compare IO definitions against physical map, highlight differences 48

49 SAN Health TM for FICON IOCP Link Addresses Node Descriptors Sequence Numbers Identify Devices 49

50 50

51 51

52 52

53 53

54 Performance and Capacity Application RMF Brocade Tools Performance Suitable Preferred Capacity Planning Preferred Suitable The Brocade tools (SANHealth, EFCM) are best for real time problem identification since they can collect data for intervals as short at 5 seconds However, RMF or RMF based SW is preferred for long term trending and data collection since RMF is the basis of you existing capacity planning process. 54

55 Intellimagic RMF Magic Effective Data Rate & Connect By plotting effective data rate and connect together it is easier to spot FICON elongation Effective Channel Data Rate (MB/s) and Connect (ms) for XP-3 Chart generated when requesting LSS-level response time charts (Breakdown charts) MB/s :00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22: msec Effective Channel Data Rate (MB/s) Connect (ms) 55

56 Channel Charts FICON Charts & Migration: can request three sets of charts (in addition to the tables previously supported) Channel busy per LPAR One line for every logical channel Channel busy per footprint (LPARs sharing channels) One line for every channel path Channel busy per DSS One line for every channel path (identified as plex:channelno ) Notes: Requires channel statistics to be loaded in database Only in available in daily view 56

57 FICON Director Charts RMF Magic can process FICON director statistics FICON Director records (74.7) are collected when: Director Microcode has been licensed FICON director reporting is enabled in RMF FICON collection in enabled in PARMLIB IECIOSxx FICON statistics should be collected on one LPAR only Processing of FICON director statistics during Reduce can be suppressed (to save processing time): COLLECT NOFCD Can skip loading of FICON director and channel statistics Run Control Center - database load page 57

58 FICON Director Charts User controls level of detail: Information about all ports or ISL only For ISL s you can chart individual port activity Chart sets include: Receive and Transmit MB/s Average receive and transmit frame size Received and Transmitted frames/sec Pacing delay (if any) Link errors (if any) Note: FCD reports do not show DSS or Host port numbers! 58

59 Port Received (MB/s) for Director MCD-28H MB/s Host Multi CU Single CU :02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 59

60 Conclusions and Observations Doubling bandwidth of a FICON channel offers no guarantee of performance gains. Much more to performance than bandwidth Before you think about 8 Gbps FICON, make certain you are getting the most performance for your investment in 4 Gbps, or even 2 Gbps FICON. Proper analysis of your FICON infrastructure from host to storage, using available tools is essential Don t fall into the classic trap of trying to improve performance by throwing more hardware at the problem without analysis. Brocade has the expertise and the tools to optimize your mainframe s FICON infrastructure from host to storage. And yes, if you need it, we have some pretty darn good hardware too! 60

61 Extra Material Following slides are some basics on buffer to buffer credits and flow control

62 Packet Flow Fundamental concepts: Prevent a transmitter from overrunning a receiver by providing real time signals back from the receiver to pace the transmitter Manage each I/O as a unique instance Buffer-to-Buffer flow control: B-to-B flow control is managed by two optically adjacent ports in the FICON I/O path Separate, independent pool of credits manages Buffer-to-Buffer flow control (BB_Credits) End-to-End flow control: E-to-E flow control manages the logical flow of control between two end nodes Intervening directors do not participate in E-to-E flow control 62

63 Buffer to Buffer Flow Control Flow control between two optically adjacent ports in the I/O path. Separate, independent pool of credits manages Bufferto-Buffer flow control (BB Credits). Transmitter count non-zero: transmitter is free to continue sending data. The FC-FS specifications allows the transmitter to be initialized at zero, or at the value of BB_Credit and either count up, or count down on frame transmit. Varies by vendor 63

64 Buffer-to-Buffer flow control (3) It takes light 5 nsec to propagate through 1 meter of optical fiber 50 µsec to travel 10 km. Faster links, longer distances leads to a performance drag similar to ESCON droop. Need BB credit values>1 and frame streaming. Frame streaming: allowing a sending port to send more than 1 frame without having to wait for a response to each. Approach 100% link utilization 64

65 Buffer Credit Concepts Define the maximum amount of data that can be sent prior to an acknowledgement Buffer credits are physical ASIC port or card memory resources and are finite in number as a function of cost Within a fabric, each port may have a different number of buffer credits The number of available buffer credits is communicated at fabric logon (FLOGI) 65

66 Buffer Credit Concepts(2) One buffer credit allows a device to send one 2112 byte frame of data (2K usable for z/os data) Assuming that each credit is completely full, you need one credit for every 1 KM of link length over a 2 Gbit fibre Unfortunately, z/os disk workload rarely produce full credits. For a 4K transfer, the average frame size for a 4K transfer is 819 bytes Hence, five credits would be required per KM over a 2 Gbit fibre 66

67 BB credit consumption tracking process Before any data frames are sent, the transmitter sets a counter equal to the BB-credit value. For each data frame sent by the transmitter, the counter is decremented by one. Upon receipt of a data frame, the receiver sends a status frame (R_RDY) to the transmitter indicating that the data frame was received AND the buffer can receive another data frame. For each R_RDY received by the transmitter, the counter is incremented by one. 67

68 B-to-B and E-to-E Control IUs BB Credits BB Credits BB Credits 4 Gbit <= 10 Gbit 2 Gbit ISL BB Credits Director BB Credits Director BB Credits Buffering in the director allows each segment to run at a different data rate. When fibre costs are high between your sites, you can trade off ISL data rate versus the higher cost of the director ports IUs 68

69 FICON Channel Performance-Large Sequential Read or Write Data Transfers Reference: IBM 69

70 Frame Pacing versus Frame Latency Frame Pacing is an FC4 application data exchange level measurement and/or throttling mechanism Pacing is determined by the channel based upon feedback from the control unit It is managed between the server and storage or channel and control unit It is gauged by the performance characteristics of the data exchanges and resource availability and can be modified by the channel on the fly Frame Latency is a frame delivery level measurement The mount of time it takes a frame to be delivered and it depends upon how it is measured Most measurements are the amount of time it takes a frame to travel from the server to the storage (channel to control unit) Each element that handles the frame contributes to this latency measurement (HBA, switch, storage HBA) Our measurement is the average amount of time it takes us to deliver a frame to the destination port (i.e. from the moment the SOF enters the source port to the moment the EOF leaves the destination port) Why do we care about these measurements? They help determine which parts of the system diminish the ability to send frames at the theoretical speed of light in fibre One way to look at latency is it's like measuring frame "friction" 70

71 FICON Open Exchange Here is a simple way to understand what an Open Exchange is all about An Open Exchange is like a Telephone Conversation: Pend Time is from picking up the phone, hearing it ringing and then someone picks up at their end and we hear a voice Disc Time is that the wrong person answered the phone and we have to wait until we hear the correct person s voice on the phone line Conn Time is having a conversation with the right person The Open Exchange time is from when we picked up the phone at our end of the conversation until the person at the other end hangs up his phone And there could be many of these conversations happening concurrently 71

72 Channel Charts - Example Actual Mbyte / Max Mbytes Host FICON card bus busy Read MB Write MB 72

73 Port Transmitted (MB/s) for Director MCD- 28H MB/s Host Multi CU Single CU 0 0:02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 73

74 Transmit Frame Size (byte) for Director MCD- 28H byte Host Multi CU Single CU 0:02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 74

75 Transmit Frame Rate (frames/sec) for Director MCD-28H frames/sec Host Multi CU Single CU :02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 75

76 Sample ISL Charts Port Received (MB/s) for ISLs for all FICON Directors MB/s :59 1:59 3:59 5:59 7:59 9:59 11:59 13:59 15:59 17:59 19:59 21:59 MCD-IN002 MCD-IN050 MCD-XK004 MB/s Port Received (MB/s) for ISLs for Director MCD-IN D B 23:59 1:59 3:59 5:59 7:59 9:59 11:59 13:59 15:59 17:59 19:59 21:59 76

77 Pacing Delay (ms) for Director MCD-28H ms Host Multi CU Single CU 0:02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 77

78 Receive Frame Size (byte) for Director MCD- 28H byte Host Multi CU Single CU 0:02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 78

79 Receive Frame Rate (frames/sec) for Director MCD-28H frames/sec Host Multi CU Single CU :02 0:17 0:32 0:47 1:02 1:17 1:32 1:47 79