Addressing a Performance issue The drilldown approach. Laurent Léturgez

Transcription

1 Addressing a Performance issue Laurent Léturgez

2 Whoami Oracle Consultant since 2001 Former developer (C, Java, perl, PL/SQL) Hadoop aficionado Data Management on Premises and in the Cloud Blogger since Twitter

3 500+ Technical Experts Helping Peers Globally 3 Membership Tiers Oracle ACE Director Oracle ACE Oracle ACE Associate bit.ly/oracleaceprogram Connect: oracle-ace_ww@oracle.com Nominate yourself or someone you know: acenomination.oracle.com

4 Agenda The Drill down approach It s [always] a question of time Active Average Session and Database load Identify Bottlenecks with various tools Qualify identified bottlenecks to reduce time consumption Various tools for a better analysis Code instrumentation PL/SQL Profiling

5 The drill down approach

6 Have you ever met this kind of user reactions? Have you ever answer them this? My Application is slow can you help me? It s slow (or it s hang) help! it must be the database?

7 Usually, we need more information When? How the problem occured? Is it general or specific use case? Any error message? Are you sure it s really the database? We need to trust our user (or interview more than one user). We have to analyze the issue by ourself.

8 Time is the key of analysis A session can spend time in different ways It waits for work to do Idle Wait time It waits for a system call or something to complete (Waiting for a lock, an IO etc.) Active Wait Time (or Non Idle Wait time) It executes oracle code on CPU DB CPU Time Active time in a session Active Wait Time + DB CPU Time Active time in Database is DB Time DB Time = σ SID=1 SID=n (Active Wait Time + DB CPU Time)

9 For a Session 0 DB Time Elapsed time 2 sec. 8 sec. 3 sec. 3,5 sec. User IO DB CPU TX contention DB CPU Time = 3 sec Non Idle wait time = ,5 = 13,5 sec DB Time = ,5 = 16,5 sec Elapsed Time = 60 sec At Database level Active Average Session AAS = Elapsed Time = 60 sec σ DBTime Elapsed Time

10 Database Time for a not overloaded system (2 CPUs) User IO DB CPU TX contention

11 Database Time for an overloaded system short period (2 CPUs) User IO DB CPU TX contention

12 Database Time for an overloaded system (2 CPUs) User IO DB CPU TX contention

13 Active Average Session is a key indicator for database load AAS = or close to 0: Database is idle AAS < # CPU Cores, no system bottleneck AAS ~ # CPU Cores, database uses all system resources (If one DB / system) AAS > # CPU Cores, Database loaded (depends on the part of CPU in AAS) AAS >> 2 x # CPU Cores, Database is overloaded Database load = AAS / # CPU Core DB Load = or close to 0: Database is idle DB Load < 1, no system bottleneck DB Load ~ 1, database uses all system resources (If one DB / system) DB Load > 1, Database loaded (depends on the part of CPU in AAS) DB Load >> 2, Database is overloaded

14 When AAS/Database load is on top? Identify AAS or Database load peak time Not very easy with AWR or Statspack Reports A little bit easier with EM top Activity page (Performance tab) Trending and Data visualisation is the solution Explore AWR tables and views (and ASH) Explore Statspack tables and views Graph and plot Heatmap Trends

15 Trending and Data visualisation of AAS Granularity matters From ASH, you can get AAS every seconds V$ACTIVE_SESSION_HISTORY (or less by modifying _ash_sampling_interval ) every 10 secs DBA_HIST_ACTIVE_SESS_HISTORY Or more. by writing the correct SQL statement From STATSPACK, you can get AAS from the time model analysis between two snapshots

16 Trending and Data visualisation of AAS Granularity matters: example AAS every hour from DBA_HIST_ACTIVE_SESS_HISTORY (Thanks Marcin Przepiórowski) SELECT TO_CHAR(sample_time,'YYYY-MM-DD HH24') mtime, round(decode(session_state,'waiting',count(*),0)/360,2) aas_wait, round(decode(session_state,'on CPU',count(*),0) /360,2) aas_cpu, round(count(*)/360,2) aas FROM dba_hist_active_sess_history GROUP BY to_char(sample_time,'yyyy-mm-dd HH24'), session_state ORDER BY mtime

17 Trending and Data visualisation of AAS MTIME AAS_WAIT AAS_CPU AAS AAS_WAIT AAS_CPU AAS

18 But my database is overloaded or not? Add CPU Core number, directly in your graph AAS_CPU AAS CORE

19 But my database is overloaded or not? Or modify your query to get the db load and plot it directly. SELECT mtime, ROUND(SUM(load),2) LOAD FROM (SELECT TO_CHAR(sample_time,'YYYY-MM-DD HH24') mtime, DECODE(session_state,'WAITING',COUNT(*),0)/360 c1, DECODE(session_state,'ON CPU',COUNT( *),0) /360 c2, COUNT(*)/360 cnt, COUNT(*)/360/cpu.core_nb load FROM dba_hist_active_sess_history, (SELECT value AS core_nb FROM v$osstat WHERE stat_name='num_cpu_cores ) cpu GROUP BY TO_CHAR(sample_time,'YYYY-MM-DD HH24'), session_state, cpu.core_nb ) GROUP BY mtime ORDER BY mtime; MTIME LOAD ,2 1 0,8 0,6 0,4 0,2 0-0, LOAD

20 Heatmap to identify bottleneck Based on previous queries and Oracle PIVOT function See: Dataviz can be done with various tools: Tableau software Microsoft Excel with conditional formatting Python with plotly library

21 Example (With Microsoft Excel)

22 Examples (with Python and Plotly)

23 So what is this drilldown approach? First identify if the database is overloaded (User Interviews, AAS wide analysis) Then, identify when the database is overloaded (Heatmap, AAS trending) Then, identify how the DB time is distributed (AAS trending) More CPU Time than Active Wait time? More Active wait time than CPU Time? Run the AWR Report or Statspack to get more details (AWR and SP reports) Reduce CPU Time or Active Wait Time or Both (with the help of your brain!!) If more CPU time, analyze SQL statements that burns buffer cache for example If more Active wait time, identify which one(s), resolve the issue(s)

24 So what is this drilldown approach? Identify if the database is overloaded (User Interviews, AAS wide analysis) Identify when the database is overloaded (Heatmap, AAS trending) identify how the DB time is distributed (AAS trending)? More CPU Time than Active Wait time? More Active wait time than CPU Time? Run the AWR Report or Statspack to get more details Reduce CPU Time or Active Wait Time or Both If more CPU time, analyze SQL statements that burns buffer cache for example If more Active wait time, identify which one(s), resolve the issue(s)

25 Ok, but if I haven t bought Diagnostics Pack or if I run a Standard edition? Heatmap not possible because it s based on ASH You can graph AAS, AAS_WAIT and AAS_CPU on a large period Then reduce time scale, redo the same AAS trending How? See: Time Model Analysis with a specific function Get DB Time and DB CPU, and calculate Active Wait Time for every period between two snapshots Calculate AAS = DB Time / Elapsed for every period between two snapshots And plot it!

26

27 Various tools for a better analysis

28 Code Instrumentation Use of DBMS_APPLICATION_INFO Add information in V$SESSION, V$SESSION_LONGOPS, V$SQL_MONITOR, V$SQL (and some others) MODULE ACTION CLIENT_INFO (Only in V$SESSION and V$SQL_MONITOR) Then dispatched to ASH (V$ACTIVE_SESSION_HISTORY, DBA_HIST_ACTIVE_SESS_HISTORY) AWR (DBA_HIST_SQLSTAT) Statpack (only Module for STATS$V_$SQLXS, STATS$SQL_SUMMARY and STATS$TEMP_SQLSTATS) Note: CLIENT_INFO is not dispatched

29 Code Instrumentation Without Code instrumentation With Code instrumentation

30 Code Instrumentation: Java sample code public static void main(string[] args) throws Exception { DriverManager.registerDriver(new oracle.jdbc.oracledriver()); // Warning: this is a simple example program : In a long running application, // error handlers MUST clean up connections statements and result sets. String module, prev_module; String action, prev_action; Connection c = DriverManager.getConnection("jdbc:oracle:thin:@ :1521:orcl", "system", "oracle"); CallableStatement call = c.preparecall("begin dbms_application_info.set_module(module_name =>?, action_name =>?); end;"); module="paycheck"; action="header"; try{ call.setstring(1,module); call.setstring(2,action); call.execute(); } catch (SQLException e) {e.printstacktrace();} // PAYCHECK HEADER EDITION HERE module="paycheck"; action="main"; try{ call.setstring(1,module); call.setstring(2,action); call.execute(); } catch (SQLException e) {e.printstacktrace();} finally {call.close();} Backup previous module and action with dbms_application_info. get_module function // PAYCHECK MAIN PART EDITION HERE } c.close();

31 Code Instrumentation : the drilldown approach Identify top module activity For top modules, identify the top Action When possible, identify client with CLIENT_INFO Complete performance analysis of the specific Module/action AWR / ASH TOP SQL identification in this module / action Code inspection Code profiling

32 PLSQL Profiling Profile runtime behaviour of PLSQL code Allow bottleneck identification Introduced in Oracle 8i Oracle 11gR1 introduced hierarchical PLSQL profiler

33 PLSQL Profiling : How does it work? DBMS_PROFILER. start_profiler DBMS_PROFILER. stop_profiler R U N T I M E

34 Real Life example #1 Night batch is too long (and ends after 4AM) Oracle 10g classic profiler The culprit is the billing (sub)-batch (PLSQL) Profiling

35 Real Life example #1 results UNIT_NAME LINE# TOTAL_OCCUR TOTAL_SEC MIN_SEC MAX_SEC /... XNPCK_XXTRACE ,42 0,02 XNPCK_XXTRACE ,88 0,02 XNPCK_XXTRACE ,86 0,01 XNPCK_XXTRACE ,86 0,04 XNPCK_XXTRACE XNPCK_XXTRACE ,28 0,02 XNPCK_XXTRACE XNPCK_XXTRACE XNPCK_XXTRACE XNPCK_XXTRACE XNPCK_XXTRACE ,55 0,02 XNPCK_XXTRACE XNPCK_XXTRACE ,23 0,03.../... XNPCK_XXTRACE XNPCK_XXTRACE ,78 0,02 XNPCK_XXTRACE /... XNPCK_XXTRACE ,42 0,01 XNPCK_XXTRACE ,6 0,01 XNPCK_XXTRACE ,83 0,01 XNPCK_XXTRACE XNPCK_XXTRACE XNPCK_XXTRACE XNPCK_XXTRACE ,2 0,02 XNPCK_XXTRACE ,52 0,02 XNPCK_XXTRACE ,83 0,02 XNPCK_XXTRACE XNPCK_XXTRACE ,54 0,03 Line 56 of package body XNPCK_XXTRACE has been executed times. Each execution took 0 to 0,02 sec The global time for this step is 1001 seconds What are these packages names??? xxtracexx After having a look in the code near these lines numbers, We found something like that: file1:= utl_file.fopen('utl_dir, debug.txt','w'); utl_file.put_line(file1, Some information' ); utl_file.fclose(file1); In a loop!!

36 PLSQL Hierarchical Profiler : How does it work? DBMS_HPROF.start_profiling (location=>dir, filename=> profiler.txt ) DBMS_HPROF.stop_profiling R U N T I M E profiler.txt

37 Real Life example #2 Customer complains about very slow application Culprit is the database (11gR2) DB Load constantly upper to 4 Customer bought new Oracle CPU, application capacity increases After a while, application becomes slow, application capacity cannot grow the same way the application becomes popular After code instrumentation, a problematic code path is identified profiling (hierarchical)

38 Real Life example #2 hierarchical profiler results SUBTREE FUNCTION ELAPSED TIME ELAPSED TIME LEVEL NAME LINE# TYPE usec usec CALLS MGUSER.PR_CALC_BULL_FIN_STC_AED 1 PLSQL /... 2 MGUSER.PR_TRAITEGEN_MULT_DEPART_CDD 1 PLSQL /... 3 MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD 1 PLSQL /... 4 MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD. static_sql_exec_line SQL MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD. static_sql_exec_line SQL MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD. static_sql_exec_line SQL MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD. static_sql_exec_line SQL MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD. static_sql_exec_line SQL MGUSER.PR_TRAITEGEN_LISTE_DEPART_CDD. static_sql_exec_line SQL /... but its sub-calls took µsec need to analyze next level (4) This call took µsec This call took µsec No more sub-execution because Subtree time = function time (and because it s a SQL op). A single execution took : / 239 = µsec Compared to other SQL in this level it s the main time consumer NAME TEXT LINE PR_TRAITEGEN_LISTE_DEPART_CDD begin 576 PR_TRAITEGEN_LISTE_DEPART_CDD PR_PERF('MILI', '','','AED'); 577 PR_TRAITEGEN_LISTE_DEPART_CDD select /*+ index(pgd PK_PARA_GENE_DNAC) */ 578 PR_TRAITEGEN_LISTE_DEPART_CDD id_para, 579 PR_TRAITEGEN_LISTE_DEPART_CDD 'PRES' as status_aed, 580 PR_TRAITEGEN_LISTE_DEPART_CDD date_gene, 581 Oh wait! a hint we analyzed the sqlplan, tune it by simply remove the hint PROBLEM FIXED!!

39 PLSQL Hierarchical Profiler : Data Visualisation Tools exist to visualise PL/SQL hierarchical profiles SQL Developer Martin Büchi tools Set of packages (Java & PLSQL) to display PLSQL Profile in a web Browser Google Developer tools: cpuprofile Brendann Greg s FlameGraph

40 PLSQL Hierarchical Profiler : Data Visualisation with Flame Graph SQL> exec ora_hprof#.flatten('work_dir','profiling_4e0f4c0a96016c63e0537a1ea8c0113f_2202','profile_flat.txt'); $ flamegraph.pl /var/tmp/profile_flat.txt > /var/tmp/profile_flat.svg profiler.txt

41 Conclusion Addressing a performance issue. Key is time analysis Proceed from the general to the detail After identifying bottlenecks, use the right tool for the right job Code Instrumentation PLSQL Profiling (and hierarchical profiler) Better when using graphical tools

42 Questions?