How To Manage Disk Effectively with MPG's Performance Navigator

Transcription

1 How To Manage Disk Effectively with MPG's Performance Navigator

2 Today s Agenda: How To Really Understand Your Disk Environment a) Managing Disk is Hard It Doesn t Have To Be b) Measure Individual ASPs c) Disk Growth - MPGLOB d) Disk Performance Best Practice Guidelines e) Internal vs External Disk How To Understand Performance New Product Functionality Major change in one of our existing Performance Scripts Q&A

3 The Essence of a Good Systems Admin / Programmer I ve been using the product now about 20 years I ve seen a lot of changes to the product over the years (Now V18!) Imagine all the problems / projects over time amazing real life examples (Think of the complexity of IT issues) Phone call from a fairly new user Expectations of the Product He asked: Why doesn t the product just tell me what the problem is? Sometimes the answers are easily found, but sometimes you have to play a detective & dig (case in point my external disk example later in this demo) Our # 1 goal is to build functionality where your everyday problem determination, performance management, and capacity planning tasks are easier All of us system admins wish we has a magic button we can push

4 MPG Makes Managing The Disk Component Easy For You Historical Graphs Current Day Graphs Service Level Graphs Detailed Disk Analysis Summary Reports / Power Analytics Reporting SSD Analysis External Disk Analysis

5 MPG Makes Managing The Disk Component Easy For You Current Day / Historical Graphs IBM Data MPG Historical Data

6 MPG Makes Managing The Disk Component Easy For You Reports & Power Analytics Spreadsheet Format HTML Output (Can Be a PDF)

7 MPG Makes Managing The Disk Component Easy For You Library / Object / IFS Graphs Entire suite of drillable graphs

8 When we are managing disk, we are concerned with two things Disk Space Utilization (i.e. Why is my disk growing) Disk Performance (Metrics: Arm Utilization, Disk Response Time )

9 Disk Growth

10 Disk Space Utilization Two Types of Growth Disk Growing Over Time Disk Growing Today (This is never good)

11 Why Is Disk Growing Today?

12 A Real Life Example At My Old Company Disk Was Having Alarming Growth / Operations Staff / Management Was Freaking

13 Disk Space Utilization The menu order speaks volumes Historical Disk Current Day Disk In theory, we are never supposed to have disk issues today

14 The $64,000 question is: What type of things would make disk space increase dramatically right now A job using temporary storage A job writing out to disk (i.e. maybe someone is copying a file from test to prod)

15 There has to be a better way

16 There has to be a better way There is!!!!!!

17 Why Is Disk Growing Today? (DEMO)

18 Why Is Disk Growing Historically?

19 Creating MPGLOB (Libraries & Objects) File Recommended method RTVDSKINF QUSRSYS/QAEZDISK Data Reduction (see types below) MPGLIB / MPGLOB Note: PERFNAVDS can be substituted for the RTVDSKINF collection process that is started via the GO DISKTASKS menu. (If used, simply put the PERFNAVDS job on hold) Note: From Joe Program MPGDRD is called in PERFNAVDR. RTVDSKINF is usually done in a submitted job. Most customers would schedule RTVDSKINF with GO DISKTASKS. RTVDSKINF is a very long running. It could run for hours. If it runs in batch, mention that it has to finish. So MPGLOB does not get updated until Perfnavdr Data Reduction Options: 1) Data Reduction Manually 2) Via normal PERVNAVDR

20 What Happens With Two ASPs? We have two ASPs 1 and 33. PN/400 is only collecting library data for ASP 1. Our ASP 33 is an independent ASP (IASP) where all of our big data libraries are...response From Joe Camilli The answer to your other questions is that there is an alternative to PERFNAVDS > MPGLIB/MPGOBJD. The alternative is RTVDSKINF, which is scheduled through GO DISKTASKS. RTVDSKINF -> QUSRSYS/QAEZDISK -> PERFNAVDR -> MPGLIB/MPGLOB. Each night PERFNAVDR checks for new information in either QAEZDISK or MPGOBJD. If you collect the iasp information with RTVDSKINF, then the nightly data reduction job, PERFNAVDR, will find it and use it. There is a parameter on RTVDSKINF to collect iasp information. This collection would be a separate job. I don't know the details of how to schedule this. I assume you can do it through GO DISKTASKS When RTVDSKINF collects iasp information, the output file is not QAEZDISK. It's something like QAEZD PERFNAVDR automatically checks for files with names like this. Also, by the way, RTVDSKINF is better than PERFNAVDS. You can hold PERFNAVDS if you're running RTVDSKINF on a regular basis. RTVDSKINF collects some extra information that I put in MPGLIB/MPGLOB. You'll notice in the file MPGLOB, some of the "libraries" are not really libraries. You'll see several "libraries" that start with an asterisk: *TMPSPC, *TDSKSPC, *STMF, etc. You can guess from the names what these are, and then from Performance Navigator you can get a report and historical graph of *STMF, for example. My understanding that the independent ASP collection cant be scheduled it an on demand request but once done, new libraries (from iasp) would indeed be in MPGLOB

21 (Disk Growing Historically)

22 (Disk Growing Historically) Disk Space Utilization Predicting What Will Happen With A Simple Mouse Click! Using Extrapolation

23 Disk Space Utilization Measuring Individual ASPs The Magic Triangle Pulldown To Make Your Job Easier Changing ASPs is easy!

24 Disk Space Utilization What Is The Best Practice Guideline? Press the button

25 Disk Space Utilization What Is The Best Practice Guideline? Press the button twice

26 Disk Space Utilization What Is The Best Practice Guideline? MPG guideline is 80% - Industry GL as well But you can set your own guidelines this customer was more conservative (70%)

27 (Disk Growing Historically) Back To Our Alarming Growth

28 Another $64,000 question: What type of things would make disk space increase dramatically over time Libraries Increasing In Size The IFS Increasing In Size (Directories, Stream Files) Temp Space Grew (Memory Leak?)

29 There has to be an easy way to understand why disk is growing historically There is!

30 Historical Disk Growth Power Analytics (DEMO)

31 Disk Performance

32 Disk Arm Utilization What Is The Best Practice Guideline? MPG recommends that the disk arm utilization be less than or equal to 15% busy

33 Disk Arm Utilization Is My Disk Arm Utilization Within The Best Practice Guideline?

34 Disk Arm Utilization Is My Disk Arm Utilization Within The Best Practice Guideline?

35 Disk Arm Utilization Is My Disk Arm Utilization Within The Best Practice Guideline? (8 am 5 pm)

36 Disk Arm Utilization Is My Disk Arm Utilization Within The Best Practice Guideline? (3 pm 5 pm)

37 Disk Arm Utilization How Has It Been Performing Historically?

38 Power of Historical Data Discovered a change with disk response time

39 Disk Response Time Is it within the best practice guideline? It was until the 9 th of October

40 Disk Response Time What Is The Best Practice Guideline? MPG recommends that the total disk response time be < 4 ms

41 Disk Response Time Is it within the best practice guideline?

42 Disk Response Time Is it within the best practice guideline?

43 Disk Response Time Is it within the best practice guideline?

44 Understanding Disk Response Time Service Time vs. Wait Time

45 Understanding Disk Response Time Service Time vs. Wait Time A Trip To The Bank! Wait Time is easy You are waiting in line for a bank teller Service Time You are waiting for the teller to complete your deposit NOTE: Just like in a bank where everyone has unique banking needs, disk requests are the same way (Some take longer than others) Think of Bank Tellers as Disk Arms The more tellers that are present, the faster people can have their banking done. The more disk arms that are present, the faster disk work can be completed.

46 Disk Response Time Good Environment vs. Bad Environment We need bank tellers (disk arms) Good Disk Resp Time Environment Bad Disk Resp Time Environment

47 What To Look For In Your Disk Response Time Graphs: Total Disk Response Time < 4 ms More Green (Service Time) Than Red (Wait Time) In other words, we want to be servicing the disk. If a lot of red exists, your disk requests are queuing

48 Is My Disk Response Time Within The Best Practice Guideline? (24 Hour View)

49 Is My Disk Response Time Within The Best Practice Guideline? (3 pm 5 pm)

50 Disk Response Time How Has It Been Performing Historically?

51 Remember this Check out the service level change Before vs After

52 Service Level Before vs After The Change

53 Audience Poll

54 External vs. Internal Disk Environment Whether you re a 35 year IT veteran like myself or a newbie to the IBMIi world Diving into the external disk arena is a new world Collection Services Data Now we are introduced to nmon data concept Because of the industry trend / change, a lost of us were forced to support this new environment with little or no training and the like Unix world is just different (not wrong) Black Hole to a lot of us

55 Disk Arm Utilization External vs. Internal Disk Environment FYI External Disk Environment Note: External disk environments use logical arms As a result, disk arm utilization can be a bit tricky For example: 10 physical arms, but you have 20 logical arms defined The disk arm % shown (12% busy) based on the logical number of arms (20) However, the real arm utilization would double because we really have 10 arms

56 Disk Arm Utilization External vs. Internal Disk Environment FYI External Disk Environment Note: The performance management rule is simple for external disk environments: Look at arm % for trends and the like, but look at disk response time for a true indicator of your disk performance

57 External vs. Internal Disk Performance / Troubleshooting Tip Internal disk has a logical to physical ration of 1 to 1. External disk hardly EVER has a logical to physical ration of 1 to 1. Why is this important? With a 1 to 1 ratio you can depend on arm utilization to determine if you have a disk performance issue or not. With a 1 to 1 ratio the net results of IO/Sec, Read to write ratio, Block size per IO and cache hits is arm utilization. With external disk we also need to look at disk response time. Disk response time is also the net results of IO/Sec, Read to write ratio, Block size per IO and cache hits. With external disk average, arm utilization is important to make sure you aren't queuing in the operating system. Remember, there is only one queue for each logical unit (LUN). As a general guideline with external disk, make lots of LUNs. Its free and it will perform better.

58 Why External Disk Analysis Can Be Tricky

59 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

60 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

61 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

62 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

63 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

64 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

65 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

66 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button)

67 Why External Disk Analysis Can Be Tricky Aka: You have to analyze many components (No magic button) It s an application issue!

68 Why External Disk Analysis Can Be Tricky #2

69 So I just discovered this on our development machine started on the 9th Discovering A Change

70 So I drilled down into the 9 th Notice the problem started after coming back up

71 So I drilled down into the 9 th Notice the problem started after coming back up

72 So lets troubleshoot a bit Let s look at our current hardware setup: Enterprise Hardware Summary

73 So lets look at our other LPARs: AIX720

74 So lets look at our other LPARs: AIX720

75 So lets look at our other LPARs: MPGSUSE (Linux)

76 So Now What Do We Do? External disk environment SO WE MUST MONITOR / INVESTIGATE THE VIOS SERVER

77 So lets look at our other LPARs: VIOS1

78 So lets look at our other LPARs: VIOS1

79 So lets look at our other LPARs: VIOS1 Data is fascinating

80 So lets look at our other LPARs: VIOS1 Data is fascinating

81 So Now What Do We Do Now? After this webinar, we are actually going to talk about what to do next We got lucky that it s our development box Can you imagine if this was happening to your production system(s)?!? The bottom line if you have external storage It s imperative that you monitor the VIOS server too That way you see the entire frame performance no more guessing!

82 New / Changed Functionality

83 Enterprise Performance Overview Change Since it s a performance report, we are now using disk response time (Removing Disk Space Utilization )

84 Enterprise Performance Overview Change Since it s a performance report, we are now using disk response time (Removing Disk Space Utilization )

85 But I Use This Functionality To Track & Understand My Disk Space Needs!!!

86 New Enterprise Disk Space Analysis One easy spot to understand your disk space needs for your entire enterprise (Wait until you see what it can do!)

87 New Enterprise Disk Space Analysis One easy spot to understand your disk space needs for your entire enterprise (Wait until you see what it can do!)

88 Disk Capacity Planning

89

90 Q & A