Intelligent Monitoring Framework (IMF)

Transcription

1 Intelligent Monitoring Framework (IMF) -- Smarter Availability Bhavin Thaker VCS Engineering March 2012 Total number of slides = 26 (Slide-deck Version 1.4.2) IMF: Intelligent Monitoring Framework 1

2 Poll How many use IMF as of today? IMF: Intelligent Monitoring Framework 2

3 Agenda 1 What s the problem? 2 Why use IMF? 3 How to use IMF? 4 How is magic of IMF done? -- Architecture 5 What is ProPCV in SFHA 6.0? IMF: Intelligent Monitoring Framework 3

4 What s the problem? IMF: Intelligent Monitoring Framework 4

5 From Polling to Asynchronous Monitoring IMF: Fast Failure Detection and Failover Traditional Monitoring Framework Intelligent Monitoring Framework Polling Asynchronous Most Clustering Solutions Poll based Monitoring Veritas Cluster Server (5.1 SP1 & 6.0) Intelligent Monitoring Faulted Faulted Resources Being Monitored Faulting Registering Resources Being Monitored IMF: Intelligent Monitoring Framework 5

6 Why use IMF? Isn t 1 minute good enough? We can do better! Eweek: Businesses loss on outage: Avg of $5000 / minute Well, sometimes a minute may seem like an hour Place your hand on a hot stove (for a minute) and it will seem to last an hour. Sit next to a beautiful lady for an hour, and it will seem to last only a minute. That is relativity." Albert Einstein. IMF: Intelligent Monitoring Framework 6

7 Show me the numbers (no-imf -> IMF) Fault detection time: (1 to 60 seconds) -> (~1 second) Performance Improvement using IMF -- % CPU usage for 800 Mounts and 800 Processes Monitoring Reliability: Traditional check after IMF notification IMF: Intelligent Monitoring Framework 7

8 Key Takeaway: Value Proposition IMF provides: 1. Fast Failover 2. Zero Polling Overhead IMF: Intelligent Monitoring Framework 8

9 What type of resources can have this cool feature? Notifications for: Process Online, Offline Mount Online, Offline Container Online, Offline: Sol/AIX IMF-aware Agents Process, Application agents CVMvxconfigd, CFSfsckd Oracle, Netlsnr, DB2, Sybase/IQ, PostgreSQL, MySQL WebLogic/Sphere/MQ, DLP Enforce, Containers (Solaris Zones, AIX WPARs) Mount based agents Mount, CFSMount agents detects FS-disabled instantaneously via IMF s integration with CFS IMF: Intelligent Monitoring Framework 9

10 How do we enable IMF? IMF enabled by default in SFHA 6.0 Granularity System-level (/etc/amftab) Type-level (hatype) Resource-level (hares) System-level Type-level Resource-level All resources All Oracleinstances Monitoring Type Online Monitoring Offline Monitoring Oracle1 Oracle2 Oracle3 Examples haimfconfig -enable (SFHA 6.0) Veritas Operations Manager VOM UI IMF: Intelligent Monitoring Framework 10

11 Important IMF Attributes Mode: IMF monitoring granularity 0 = Turn OFF IMF 1 = Turn ON Online Monitoring only 2 = Turn ON Offline Monitoring only 3 = Turn ON Both MonitorFreq : When to run Traditional Monitoring MonitorFreq = 10 (IMF is ON) MonitorInterval = 1 min (IMF is OFF) OracleAgent: Level-2 Monitoring (aka In-depth) always runs if enabled IMF: Intelligent Monitoring Framework 11

12 Two ways to make Agent IMF-aware VCS Agent Framework C/C++ entry points provided by Agent Script-based IMF-plugin AMF binaries libvxamf AMF driver IMF: Intelligent Monitoring Framework 12

13 3 new IMF Entry Points for Agents VCSAgIMFInit Initialize the Agent with IMF (start driving the car) VCSAgIMFRegister (lemme know when we reach there) Register a Resource with IMF; called once for every resource that should use IMF VCSAgIMFGetnotification Block in IMF for Resource State Change notifications (hey, we are there now!) IMF: Intelligent Monitoring Framework 13

14 What s the diff between IMF and AMF? AMF Binaries amfinit amfregister amfgetnotification amfconfig amfstat Script-based IMF-plugin(Perl) imf_init imf_register (uses XML config) imf_getnotification Userland OS Kernel Script-based IMF-Plugin AMF binaries libvxamf AMF driver IMF: Intelligent Monitoring Framework 14

15 Script-based IMF Plugin: Example Create the amfregister.xml file in /opt/vrtsvcs/bin/ezimfagent <xml> <Register> <RegType>PRON</RegType> <PronOptions>IGNORE_ARGS IGNORE_PATH</PronOptions> <ProcPattern>ezimf.sh</ProcPattern> </Register> <Register> <RegType>PROFF</RegType> <Owner>root</Owner> <Path>/bin</Path> <arg0>sh</arg0> <args>/opt/vrtsvcs/bin/ezimf/ezimf.sh</args> </Register> </xml> IMF: Intelligent Monitoring Framework 15

16 Script-based IMF Plugin: Example (cont d) 1) Create the file amfregister.xml in /opt/vrtsvcs/bin/ezimfagent 2) Create sym-links in /opt/vrtsvcs/bin/ezimfagent to IMF entry-points ezimfagent -> /opt/vrtsamf/imf/script51agent imf_init -> /opt/vrtsamf/imf/imf_init imf_register -> /opt/vrtsamf/imf/imf_register imf_getnotification -> /opt/vrtsamf/imf/imf_getnotification 3) Configure IMF in VCS config files: /etc/vrtsvcs/conf/config/types.cf, main.cf type ezimf ( static int IMF{} = { Mode=3, MonitorFreq=10, RegisterRetryLimit=3 } static str IMFRegList[] = { } ) That s it Your Script Agent is now IMF-enabled! Visit the IMF lab session! IMF: Intelligent Monitoring Framework 16

17 AMF Driver Terminology Reaper = VCSOracleAgent Looks Confusing? Ok, Let s take an example AMF Group = ora_db R_RID = 20 Reaper ResourceID RID = 51 PID = 1001 Ora_pmon RID = 52 PID = 1002 Ora_lmon AMF Driver RID = 53 PID = 1003 Ora_smon RID = 54 PID = 1004 Ora_lmd RID = ResourceID IMF: Intelligent Monitoring Framework 17

18 amfstat screenshot (Process) Registered Reapers (2): ======================= R_RID PID MONITOR TRIGG REAPER VCSOracleAgent... Process ONLINE Monitors (4): [ONLINE MONITORING: Node-A] ============================ RID R_RID PID GROUP ora_db ora_db ora_db ora_db RID AMF internal unique identifier for an event R_RID RID of reaper Process OFFLINE Monitors (4): [OFFLINE MONITORING: Node-B] ============================= RID R_RID PATH ARGV0 ARGS UID EUID GID EGID GROUP CONTAINER PROJECT ACTION 12 1 oracle ora_pmon_amftest <none> <any> <any> <any> ora_db global <any> Allow 13 1 oracle ora_smon_amftest <none> <any> <any> <any> ora_db global <any> Allow 14 1 oracle ora_lgwr_amftest <none> <any> <any> <any> ora_db global <any> Allow 15 1 oracle ora_dbw0_amftest <none> <any> <any> <any> ora_db global <any> Allow IMF: Intelligent Monitoring Framework 18

19 amfstat screenshot (Mount) Registered Reapers (2): ======================= R_RID PID MONITOR TRIGG REAPER VCSMountAgent... Mount ONLINE Monitors (1): [ONLINE MONITORING: Node-A] ========================== RID R_RID FSTYPE DEVICE MOUNTPOINT GROUP CONTAINER 24 0 vxfs /dev/vx/dsk/oradg_246_247/oravol /oradata mnt_ora <none> Mount OFFLINE Monitors (1): [OFFLINE MONITORING: Node-B] =========================== RID R_RID FSTYPE DEVICE MOUNTPOINT GROUP CONTAINER vxfs /dev/vx/dsk/oradg_246_247/oravol /oradata mnt_ora <none> IMF: Intelligent Monitoring Framework 19

20 IMF Agent Architecture: Events Timeline 5. Run Traditional Monitor 2. VCSAgIMFGetnotification 4. Notification 1. VCSAgIMFInit BLOCKED IMF Thread Agent Start 3. Run Traditional Monitor Agent Main Thread 4. VCSAgIMFRegister IMF: Intelligent Monitoring Framework 20

21 AMF Driver Architecture: Events Timeline USER SPACE Initialization Registration Getnotification libvxamf Setup Reaper Insert event in hash table BLOCK Lookup event in hash table AMF Driver OS Hook kernel for Reaper death notification Hook kernel for event notification Notification from kernel KERNEL SPACE IMF: Intelligent Monitoring Framework 21

22 Reactive -> Proactive Prevention of Concurrency Violation: ProPCV Detection of Concurrency Violation Detection Prevention of Concurrency Violation NEW Reactive App App App App Proactive Concurrency Violation N1 (Active) N2 (Standby) N1 (Active) N2 (Standby) VCS proactively prevents CV Default OfflineMonitoring = 5 min Reactive: Data Corruption possible! Application NOT allowed to startup Group-level attribute Off by default IMF: Intelligent Monitoring Framework 22

23 amfstat screenshot for Pro PCV Process OFFLINE Monitors (3): ============================= RID R_RID PATH ARGV0 ARGS UID EUID GID EGID GROUP CONTAINER ACTION /usr/sbin/who /usr/sbin/who <none> <any> <any> <any> <any> group_xyz <none> PREVENT /usr/sbin/nscd /usr/sbin/nscd -x -t <any> <any> 0 <none> Global ALLOW /orcl/oracle ora_lmon_busdb <none> 20 <any> 20 <any> ORCL Global ALLOW IMF: Intelligent Monitoring Framework 23

24 References Screen-cast on IMF A blog on IMF Whitepaper on IMF VCS User's and Agent Developer s guide (search for: IMF) IMF: Intelligent Monitoring Framework 24

25 Summary: Key Takeaway IMF provides: 1. Fast Failover 2. Zero Polling Overhead IMF: Intelligent Monitoring Framework 25

26 Failure is NOT an option -- NASA, Fast Failover is -- with VCS, Use IMF! Thank you! Slide Credits: Bhavin Thaker Vishal Thakur Abhijit Toley Hema Venkataramani IMF: Intelligent Monitoring Framework 26