How method-based problem diagnosis can cut downtime by 97%

Size: px

Start display at page:

Download "How method-based problem diagnosis can cut downtime by 97%"

Wilfrid Hamilton
5 years ago
Views:

1 How method-based problem diagnosis can cut downtime by 97% Presented by: Paul Offord Development Director, Advance7 Chairman of the itsmf Problem Management SIG

2 Routing of Problems

Grey Problem Characteristics Q1 Single Incident 80-90% Change-related Cause Software Failure Hardware Failure Operations Error Misconfiguration User Error Q3 Operations Error Component Known

3 Grey Problem Characteristics Q1 Single Incident 80-90% Change-related Cause Software Failure Hardware Failure Operations Error Misconfiguration User Error Q3 Operations Error Component Known Component Unknown Intermittent Hardware Failure Q2 Known Error Poor Application Logic Intermittent Software Failure Recurring Incident Q4 Intermittent Application Error Transient Overload Intermittent Infrastructure Error Incorrect Failover Operation Grey Problems are typically: What forms the bulk of ongoing recurring problems Medium or low priority (high impact / low urgency) Slow to resolve Heavy on IT resource A sign of problems to come What gets remembered What defines IT performance Additional impact: Close down IT options Create a fog Business adjusts

4 Grey Problems?

Problem Phases Phase 1 Dealt with by Help Desk / 1st line support Simple problems and user errors Service recovery based on knowledge and procedures Service recovery within 16 hours worked Phase 2

5 Problem Phases Phase 1 Dealt with by Help Desk / 1st line support Simple problems and user errors Service recovery based on knowledge and procedures Service recovery within 16 hours worked Phase 2 Dealt with by 2nd / 3rd line support Issues caused by faults, overload or misconfiguration Service recovery or fixed knowledge, tools and knowledge-base access Achieved within a further 24 worked hours Phase 3 Incident Problem or Major Incident Dealt with by 3rd line support with supplier product specialist Complex problems, often performance related and/or intermittent Fixed through pattern method, detailed product knowledge and advanced tools Fixed within further 24 worked hours Phase 4 More and more people get involved Complex problems that have dropped through Phase 3 with root cause unknown Attempts to fix through holistic method, gut feel, random upgrades, etc. May be fixed within from 10 days to 2+ years or may never be fixed

6 The ITIL Solution

7 Problem Diagnosis Gap Procedural Skills Gap Technical Skills

8 Prove you are not the cause Typical Resulting Model Prove you are not the cause Prove you are not the cause Prove you are not the cause

9 Bridge The Gap RPR Procedural Skills Technical Skills

No Enter from PM Process Gain detailed & accurate understanding of problem symptoms Choose one symptom to investigate Gain accurate understanding of the symptom environment Share, gather, explain &

Yes Analyse the captured diagnostics Root Cause identified? Yes Translate diagnostic data and present to the Support Team owning the RC technology Adequate diagnostics?

10 No Enter from PM Process Gain detailed & accurate understanding of problem symptoms Choose one symptom to investigate Gain accurate understanding of the symptom environment Share, gather, explain & sort Agreed understanding? No No No Agree diagnostic objective and plan capture of definitive data Execute the diagnostic capture plan Review the captured diagnostics Is Quality Acceptable? Yes Analyse the captured diagnostics Root Cause identified? Yes Translate diagnostic data and present to the Support Team owning the RC technology Adequate diagnostics? Yes Work with the owning Support Team to determine the fix Implement the fix and re-activate the diagnostic capture plan Fixed? Yes Exit to PM Process No No Analyse captured data No New Root Cause? Yes RPR - What to Do & How to Do It Core Process Supporting Techniques Discover - Gather & review existing information - Reach an agreed understanding Investigate - Create & execute a diagnostic plan - Analyse & iterate if necessary - Identify Root Cause Fix - Translate diagnostic data - Determine & implement fix - Confirm Root Cause addressed

11 Core Process

12 Definitive Diagnostics Direct diagnostic correlation - User experience Diagnostic events Statistical data typically of little use - Lack of time correlation - Detail lost in averaging Timestamped diagnostic event recording needed - Diagnostic logs - Network traces - Process traces - Web logs - etc.

13 When to Change Tack Root Cause Workaround Live with it Problem Fix Giveaway terms: We re just going to try one more thing We made a change and it s improved a bit

14 Illustration of Definitive Diagnostics

15 Sports Coaching System It s slow

16 Fingers Were Pointing Must be a network problem because other branch applications are slow Must be a database problem because we ve had other similar problems DBA said that it can t be the database as all of the stored procedures profiled

17 The Sports Coaching System is Slow Actual Problem Click on Appointments in the menu bar it intermittently takes 10+ seconds to respond

18 Interpreting Wiggly Graphs 100% 100% 90% 90% 80% 80% 70% 70% 60% 60% 50% Utilisation 50% Utilisation 40% 40% 30% 20% 10% 0% 11:00:00 11:02:00 11:04:00 11:06:00 11:08:00 11:10:00 11:12:00 11:14:00 11:16:00 11:18:00 11:20:00 11:22:00 11:24:00 11:26:00 11:28:00 11:00:00 11:02:00 11:04:00 11:06:00 11:08:00 11:10:00 11:12:00 11:14:00 11:16:00 11:18:00 11:20:00 11:22:00 11:24:00 11:26:00 11:28:00 11:30:00 11:32:00 11:34:00 11:36:00 11:38:00 11:40:00 11:42:00 11:44:00 11:46:00 11:48:00 11:50:00 11:52:00 11:54:00 11:56:00 11:58:00 30% 20% 10% 0% 11:30:00 11:32:00 11:34:00 11:36:00 11:38:00 11:40:00 11:42:00 11:44:00 11:46:00 11:48:00 11:50:00 11:52:00 11:54:00 11:56:00 11:58:00 Time of Day Time of Day CPU Utilisation Utilisation 11:00:00 11:02:00 11:04:00 11:06:00 11:08:00 11:10:00 11:12:00 11:14:00 11:16:00 11:18:00 11:20:00 11:22:00 11:24:00 11:26:00 11:28:00 11:30:00 11:32:00 11:34:00 11:36:00 11:38:00 11:40:00 11:42:00 11:44:00 11:46:00 11:48:00 11:50:00 11:52:00 11:54:00 11:56:00 11:58:00 Not Definitive Can t Correlate with the user experience Time of Day

19 Multi Trace Correlation

20 Time Accounting Time Contribution (s) User PC WAN Two database stored procedure calls: dbo.usp_gr_sel_stafflistwithappointmentbydate dbo.usp_gr_sel_appointmentlistbydate Web Server - Central DB Branch DB Other Application Shouldn t be there

21 A Supporting Technique The Whiteboard

22 TopFund Scenario T Giant Finance It s unusable Acme ASP

23 Whiteboard example Symptoms Topfund users get type-ahead delays Giant users get slow web access Boundaries Only users in Cambridge No other Acme users get the problem Only happens with > 1 user Other Observations Cambridge users get no other problems No problems with other 3 rd party apps No one else has problems with TopFund Possible Causes User PC Cambridge LAN Corporate WAN Southampton LAN VPN Derby LAN Citrix Servers Action Plan Trace at user PC, Cambridge router, S ton router, VPN S ton firewall, VPN Derby firewall, Citrix servers That doesn t explain why. is banned

24 TopFund Analysis Minor fault here Major fault here T Giant Finance Acme ASP

25 Putting It Into Practice A Workable Model

26 Use RPR to Create 4 th Line Support Fast & Effective Communication End to End Approach RPR Supporting Techniques

27 Examples Fund manager had an application start-up problem impacting 1,200 users for 8 months Root cause identified in 2.5 days with RPR Fixed in 3.2 days Bank had a system problem impacting 900 offshore users for 13 months Customer ran 1.2m SIP for 10 months Root cause identified in 1.5 days with RPR Fixed in 4 days Mortgage company users suffered intermittent Citrixhosted application failures for 11 months Root cause identified in 5 days with RPR (thru REACT) Downtime cost - 271k REACT cost 9,800 => payback 12.5 days

28 Key Features & Benefits Based on definitive diagnostics - Easily deals with intermittent & transient problems Inherently evidence based - Bad information has little or no impact Spans technology silos - Deals with Service problems Deterministic process - You regain control, customer regains confidence Initial objective is Root Cause Identification - It s consistently faster

29 How You Can Benefit Now! Information Wikipedia We re just going to try one more thing Ban That doesn t explain why

Fully Optimize FULLY OPTIMIZE YOUR DBA RESOURCES

Fully Optimize FULLY OPTIMIZE YOUR DBA RESOURCES IMPROVE SERVER PERFORMANCE, UPTIME, AND AVAILABILITY WHILE LOWERING COSTS WE LL COVER THESE TOP WAYS TO OPTIMIZE YOUR RESOURCES: 1 Be Smart About Your Wait