Hands-On Lotus Domain Monitoring. AdminCamp June 2012, in Gelsenkirchen Daniel Nashed, Nash!Com

Transcription

1 Hands-On Lotus Domain Monitoring AdminCamp June 2012, in Gelsenkirchen Daniel Nashed, Nash!Com

2 About the presenter Daniel Nashed Nash!Com - IBM/Lotus Advanced Business Partner/ISV Member of The Penumbra group an international consortium of selected Business Partners pooling their talent and resources focused on Cross-Platform C-API, Domino Infrastructure, Lotus Traveler, Administration, Integration and Troubleshooting Platform Focus: W32, xlinux, zlinux, AIX and Solaris Regular speaker at International Conferences DNUG Enthusiast IBM Lotus Domino Infrastructure Author of the Domino Start Script for Linux and Unix nsh@nashcom.de

3 Agenda Introduction / other Log Options Domino Domain Monitoring (DDM) Hands On DDM Q&A Any time

4 Important Server Setting notes.ini EVENT_POOL_SIZE= Set this value for any server Not just if you use DDM But it is more important when you user DDM probes Size is in Bytes The maximum size is 100 MB

5 Log.nsf and Events Domino generates errors and messages Error Object Store Manager: File does not exist Message Index update process started All logged errors and messages are raised as events Log.nsf records errors and messages sequentially Log.nsf Pro: Great for maintaining a record of all errors and messages Pro: Great for debugging, if you know what you are looking for Con: A lot of unfiltered data

6 Event Monitoring Event Monitoring has been introduced in Notes 4.5 Events and Event Task is the foundation below all logging Events4.nsf contains all log message including categorization and severity Event Monitoring can generate own events like statistic events Event Handlers can be still useful for automation Mutliple trigger types: Log To database, Run an Agent, Run Task, SNMP Trap. Send via Mail, Log to system console,... Most of the functionality is now available in DDM But Events are still the base used for log messages consolidated in DDM DDM configuration is in events4.nsf

7 Domino Domain Monitoring (DDM) Comprehensive Monitoring ddm.nsf contains focused monitoring results Detailed error messages including names of resources Suggestions for problem solution including actions! Based on the foundation build by event monitoring Event categorization and severity defined in events4.nsf Additional build in probes into the code Replication (detailed reporting for failing replication) Agent Manager (long running agents, high memory/cpu usage,...) You can also leverage statistics and platform statistics

8 Log/Event Monitoring vs. DDM As of Domino 7, all events are cached and tracked by DDM Pro Con DDM.NSF is the on-disk version and superset of the event cache DDM.NSF records a set of associated events into a single report document (not just a single event) great for recording problem context great for tracking and organizing problems great for exposing knowledge about problems great for resolving problems lousy for tracking sequential order of problems lousy for extracting ad hoc data from reports (use statistics and statrep.nsf for ad hoc data)

9 The Five C s of DDM Consolidate Track multiple, related errors in a single event report Check Assess health of functional areas Collect Access all domain wide reports from a single database Correlate Locate related reports from other servers Correct Assess knowledge base of explanations and recommendations. Click to resolve issues.

10 DDM Enhancements in Domino 8 DDM has been introduced in Domino 7 Already great options available DDM 8 is a fit and finish of what has been introduced in D7 More and enhanced views More options for corrective actions Common actions More probes More possible solutions and corrections... and many more details More features planned for next releases

11 Event Reports Like log.nsf, all events are recorded into ddm.nsf Unlike log.nsf, each set of events has it s own report document Each set of events has an associated unique ID (a PUID) Every event with the same PUID is mapped into the same ddm.nsf report document A PUID is built using the enhanced event context data Target database, target user, target agent, etc. DDM.nsf Reports are presented in a variety of views

12 DDM Architecture

13 Simple and Enhanced Events Events can have one or two associated errors. When there are two errors, the root cause is the second error. 1 error: Event: Could not locate view 'xyz' 2 errors: Object Store Manager: File does not exist Simple events are legacy events which include these attributes Time stamp, originating server, resource strings and IDs, severities, types Enhanced events also include the following attributes New types, subtypes, target server Enhanced events include one or more of these attributes Target database, target user, target UNID, extra target data (i.e. string blobs), rich text, call stacks, correlation codes, and the NOTEID of a config doc responsible for firing the event. This target data is some of the new event context

14 Event Report State Open = The event has a non-normal severity and has not been manually closed Auto Clearing Events = Some events can be automatically closed by DDM if the problem is reported as resolved with a Normal severity event Many events must be closed manually Simple events never auto close (only enhanced events) Closed = The event has a Normal severity or has been manually closed Automatically reopened if a severity change is detected Permanently Closed = Used by Admins to say I don t care about this problem, keep it out of my sight. Not automatically reopened, on a Normal severity, but continue to be tracked and recorded in its reports

15 DDM Probe Configuration Probes are used to automated check your environment There are different kind of probe types Probes are already defined in the configuration Most probes just need to be enabled Some of the probes need to be configured

16 DDM Probe Configuration Three flavors of probes (some probes have more than one): Scheduled probes run according to a configurable schedule and defaults are supplied Embedded probes run inside feature area and catch problems/issues as they occur Listening probes run when particular error codes are logged Probe configuration is quick and flexible: Defaults probe configuration documents supplied for out-of-box values Special target servers concept allows out-of-the-box probing without having to specify named servers Thresholds and result content are highly customizable what the probe will actually check probe sensitivity (when will they generate an event) what severity event the probe will generate Schedule is highly customizable for schedulable probes Probes can be enabled/disabled per server/server group

17 Probe Categories There are different probe categories with various individual probes

18 DDM Filters Event filters control what and how much information is reported to ddm.nsf. Why? Initial flood of events is striking, many of which have always been there Over time, administrators will want to adjust the volume, seeing more or less of certain events Enhanced and simple events can be filtered. Filters can target specific servers and filter out events by event type/area and severity A default filter is supplied and enabled for simple events to reduce the initial noise.

19 Server Collection Hierarchy Used to control replication of DDM events in your environment Based on automated selective replication formulas You can have multiple Server Collection Hierarchies based on your Notes Server Topology and Administration Topology The collecting server will receive all DDM events for his servers Helps to further focus information into a single instance of DDM Easy to setup once you are familiar with the configuration dialogs

20 Advanced Event Handlers Run an Agent when events occur In Memory copy of Event Document is passed in the document context It's not written into statrep after agent is exectured Sample Agent Dim session As New NotesSession Dim doc As NotesDocument Dim eventtext As String Set doc = session.documentcontext Print "EventText : >" + doc.eventtext(0) + "<" Print "ErrorCode : >" + doc.errorcode(0) + "<"

21 Advanced Event Handlers Run a Program Parameters are passed to the command-line Sample Parameter List All options enabled in event handler Process name: >C:\Lotus\Domino85\nnsharg.EXE< NshArg: Arg[0]: >C:\Lotus\Domino85\nnsharg.EXE< NshArg: Arg[1]: >7< NshArg: Arg[2]: >3< NshArg: Arg[3]: >02/25/ :00:00 CET< NshArg: Arg[4]: >Event Monitor0x33C5< NshArg: Arg[5]: >CN=nsh-d85-win-01/OU=Srv/O=NashComLab< NshArg: Arg[6]: >Access to server nsh-d85-win-01/srv/nashcomlab is slow. < NshArg: Arg[7]: >CN=nsh-d85-win-01/OU=Srv/O=NashComLab< NshArg: Arg[8]: >NAMES.NSF< OSTerm> Terminating, name: C:\Lotus\Domino85\nnsharg.EXE

22 Lotus Domino Statistics Valuable resource of information Combines Domino Statistics and Platform statistics Platform statistics depend on the OS platform but are sort of unified between platforms Check events4.nsf for a description of each platform stat per platform You should collect Server stats at least every 15 minutes (default is 90 minutes) Enable collect task, configure settings in events4.nsf Configure statistic events for important stats with the right thresholds Keep long term data to compare current and historic data You can also leverage SNMP to query stats Limitation: Only works for 1 partition per OS instance on all platforms

23 Top Statistics NSF Buffer Pool Used for Buffering Database I/O Check Server Stats Database.Database.BufferPool.Maximum.Megabytes Database.Database.BufferPool. PercentReadsInBuffer Interpretation Bad < 90% < PercentReadsInBuffer < 98% < Perfect Tune: notes.ini NSF_Buffer_Pool_Size_MB=n (in MB) Default: 512 MB

24 Top Statistics NSF Cache Used for Caching Open Databases Check Server Stats Database.DbCache.HighWaterMark Database.DbCache.CurrentEntries Database.DbCache.MaxEntries Database.DbCache.OvercrowdingRejections Interpretation Good = HighWaterMark < MaxEntries Good = 0 OvercrowdingRejections Tune: notes.ini NSF_DbCache_MaxEntries = n Default: NSF_BUFFER Pool size multiplied by 3

25 Top Statistics (Cluster) Replication Use to check Cluster Replicator Performance Check Server Stats Replica.Cluster.Failed Replica.Cluster.SecondsOnQueue Replica.Cluster.WorkQueueDepth Interpretation Perfect < 10 < SecondsOnQueue > 15 > Bad Perfect < 10 < WorkQueueDepth > 15 > Bad Tune: Add more cluster replicators optimize cluster server usage (e.g. Split active users between cluster mates)

26 Top Statistics Transactions Use for Indication of Server Load Check Server Stats Server.Trans.PerMinute Interpretation: Heavy < 30 < Trans.PerMinute (per User) > 10 > Light Tune: Analyze Heavy users and try to avoid load

27 Top Statistics Concurrent Tasks Use to checksimultaneous Active Database Connections Check Server Stats Server.ConcurrentTasks Server.ConcurrentTasks.Waiting Interpretation Waiting should be ZERO Tune: Server_Pool_Tasks = n ( e.g. 80) Server_Max_Concurrent_Trans = m (e. g. Server_Pool_Tasks * Number of Ports)

28 Top Statistics Platform Memory Used to check Allocated using memory pools and sub-allocations Check Server Stats Mem.Allocated Mem.Allocated.Process Mem.Allocated.Shared Interpretation Memory Leaks when increasing over days / weeks Tune By several parameters (bufferpool, cache, namelookup Note Be careful interpreting this statistic Not all memory might be included

29 Top Statistics Platform CPU Used to check CPU Utilization on Server Check Server Stats Platform.System.PctCombinedCpuUtil Platform.System.PctTotalPrivilegedCpuUtil Platform.System.PctTotalUserCpuUtil Interpretation: OK < 90% CombinedCpuUtil > 90% > TOO HIGH Tune Many Root Causes Possible

30 Top Statistics Paging File Use to check Server Memory Swapping to Disk Check Server Stats Platform.PagingFile.Total.PctUtil Interpretation OK < 0% < PctUtil.Avg > 10% > BAD Tune OS Level tuning, Check Memory

31 Top Statistics Platform Disk Used to check Disk Performance Check Server Stats Platform.LogicalDisk.1.AvgQueueLen Platform.LogicalDisk.1.PctUtil Interpretation Good < 2% < AvgQueueLen > 5% > BAD Good = PctUtil < 80% Tune By several parameters (bufferpool, cache, namelookup Note Platform.LogicalDisk.1.AssignedName=C points to the disk

32 Q&A I hope you enjoyed the workshop Questions now or later? Contact nsh@nashcom.de