Instrumentation An Essential Ingredient for Performance Tuning

Transcription

1 Instrumentation An Essential Ingredient or Perormance Tuning Chris Roderick, BE/CO IT/DB Oracle Lectures, June

2 Accelerator Logging Service Overview TIM ~ Signals ~ 16 data loading processes ~ 5.4 billion records per day ~ 275 GB per day à 100 TB per year throughput RAMSES > 500 registered users > 100 custom applications > 3 million extraction requests per day 7 Days raw data FGC BCT BIC PL/SQL API MK BLM BETS MDB MS BPM Rad VAC Data User> Data User> Data User> Data User> Extraction API PL/SQL iltered data transer PIC Cryo VAC PL/SQL API WIC CIET SU >20 Years iltered data LDB QPS Coll Exp CNGS ~ signals ~ 300 data loading processes ~ 4 billion records per day ~ 140 GB per day à 50 TB per year stored Data Providers Data Persistence Data Consumers Equipment DAQ FEC Equipment DAQ PLC 2

3 Mission Critical Service! Lots o Data!! Lots o Users!!! 3

4 Be wary o the users! 4

5 Big dierence between how: You think users will read/write data Users tell you they intend to read/write data Users actually read/write data 5

6 Accelerator Logging Service Lots o Data! 6

7 System misuse can have severe negative impact on stability 7

8 Instrument everything! 8

9 What is Instrumentation? Capturing inormation about system activity in real time (& over time) Who? What? Where? How? How Long? 9

10 We know what the system is doing We know how the system is perorming Throughput & Response times 10

11 Oracle is heavily instrumented out o the box ASH (using v$session, v$session_wait - real time) AWR (constant data capture over time) ADDM (analysis o AWR data) Tracing (on demand, per session, to ile) OEM (exposing all o the above in a nice GUI) 11

12 Oracle instrumentation does not include contextual inormation Already oreseen MODULE, ACTION, CLIENT_INFO DBMS_APPLICATION_INFO.SET_MODULE ( module_name IN VARCHAR2, action_name IN VARCHAR2); DBMS_APPLICATION_INFO.SET_ACTION ( action_name IN VARCHAR2); DBMS_APPLICATION_INFO.SET_CLIENT_INFO ( client_ino IN VARCHAR2); Non-transactional Minimal perormance impact (ms) 12

13 DBMS_APPLICATION_INFO in action Set via DBMS_APPLICATION_INFO 13

14 Accelerator Logging Service BigBrother Java package DBMS_MONITOR.SERV_MOD_ACT_STAT_ENABLE( IN VARCHAR2, module_name IN VARCHAR2, action_name IN VARCHAR2 DEFAULT ALL_ACTIONS); service_name BIG_BROTHER PL/SQL package DBMS_APPLICATION_INFO DBMS_MONITOR.SERV_MOD_ACT_TRACE_ENABLE( service_name IN VARCHAR2, module_name IN VARCHAR2 DEFAULT ANY_MODULE, action_name IN VARCHAR2 DEFAULT ANY_ACTION, waits IN BOOLEAN DEFAULT TRUE, binds IN BOOLEAN DEFAULT FALSE, instance_name IN VARCHAR2 DEFAULT NULL); provides transparent hooks to other uses: audit trails (custom logging, proiling) session tracing (via DBMS_MONITOR) resource management (using Resource Manager) access control (e.g. VPD / RLS) Accurate instrumentation in 3-Tier environment 14

15 Accelerator Logging Service BigBrother Java package BIG_BROTHER PL/SQL package DBMS_APPLICATION_INFO provides transparent hooks to other uses: audit trails (custom logging, proiling) session tracing (via DBMS_MONITOR) resource management (using Resource Manager) access control (e.g. VPD / RLS) Accurate instrumentation in 3-Tier environment 15

16 Database Instrumentation Example Server API instruments actions using BigBrother calls to BIG_BROTHER. CERN Accelerator Logging Service Spring HTTP Remoting 10g AS RMI Servers Spring Remoting JDBC TS Data JDBC TS Data MDB TIMBER JDBC TS Data JDBC TS Data LDB Spring HTTP Remoting Spring Remoting JDBC Metadata JDBC Metadata metadata Client API obliges provision o username/application name. BIG_BROTHER package wraps calls to DBMS_APPLICATION_INFO. Automatically picks up IP/host. 16

17 17

18 What about speciic Perormance Instrumentation? good_perormance = ((time_spent / work) <= user_expectation) Track context speciic system workload Vs time spent, over time: Measure work being done Time taken to do it Record the results (details & aggregates) 18

19 Accelerator Logging Service Who - username What - module, action Where - IP, host How - application, method, parameters How long - elapsed time or every action Contextual Inormation (dimensions aecting perormance) e.g. How many records How many signals Time window data manipulations or ilters Operation type - INSERT or MERGE (4x slower) 19

20 Data Loading Instrumentation Example Aggregates details, stores as hourly time series Batches o data sent every 30 MDB Data Writing API Logging Client JAPC Monitoring Capture details o every send: #records, #variables, datatypes, time spent, etc. Buer in memory Write in batches every 5 FGC BCT BIC MK BLM BETS MS BPM Rad VAC Accessible via TIMBER Equipment DAQ FEC 20

21 Measurements to Logging Transer 7 Days raw data ~ 95 % data reduction Millions o records per minute >20 Years iltered data MDB PL/SQL API LDB 8x jobs running every 5 PL/SQL - bulk read data by category - apply custom ilters - bulk transer 21

22 MDB to LDB Transer Instrumentation Capture details o every process execution #records read, #records transerred, time spent, etc. Keep details during 7 days Hourly Aggregates o results Store as time series Accessible via TIMBER 22

23 Data Extraction Instrumentation Example Server API uses generic Tracing Component to instrument CERN Accelerator Logging Service actions & measure perormance. Delayed batch writes o results to DB, analysis via APEX app. 10g AS RMI Servers Spring HTTP Remoting Spring Remoting JDBC TS Data JDBC TS Data MDB TIMBER JDBC TS Data JDBC TS Data LDB Spring HTTP Remoting Spring Remoting JDBC Metadata JDBC Metadata metadata 23

24 Accelerator Logging Service Many Perormance Enhancements over the years All based on understanding o system usage - through instrumentation 24

25 Conclusions? 25

26 Good Instrumentation... Facilitates Perormance Tuning, Scalability Planning, & Diagnostics Enhances System Stability Enables rapid reactive & proactive Support 26

27 Questions? Additional Use Cases Code Examples DBMS_APPLICATION_INFO wrapper Generic Java Tracing Component 27