Advanced Debugging with the System Profiler. Rennie Allen Cisco Field Application Engineer

Transcription

1 Advanced Debugging with the System Profiler Rennie Allen Cisco Field Application Engineer

2 What is the System Profiler? The System Profiler is a logic analyser for your software system The kernel records every operation in an internal buffer A tool is used to extract this data to a file The file is loaded into the analyser (i.e. the System Profiler) An array of utilities (known as views in Eclipse-speak) are used to mine the data for information about the problem Just like a hardware logic analyser: It is probably not the first tool to be selected It requires a deep understanding of both the tool, and the system being analysed When used correctly, is the only tool that can solve nearly any class of problem

3 Acquiring a Trace How to capture an effective trace

4 Classes of Problems that can be Analyzed Singleton Cyclical Sporadic Temporal Logical Logical Worst-case scenario No known source No information on type of problem Slightly better Can pinpoint when problem occurs Not reproducible Ideal scenario Can reproduce sequence of events/set of logic conditions Ideal scenario Can reproduce sequence of events/set of logic conditions The first step to analyzing a problem, is to develop a strategy to collect a trace As seen can be seen in the above diagram, there are two groupings of the above problem types that inform the basic direction of the trace acquisition strategy, and that is whether the event is singleton, or cyclical

5 Developing a Successful Trace Acquisition Strategy with the System Profiler Developing an effective trace acquisition strategy is likely the most important step Typically each problem requires development of a unique analysis strategy In order to develop a successful analysis strategy the tools/techniques available must be understood, some of these are: Linear tracing Ring-mode tracing Static filtering Dynamic filtering Dynamic statistics Dynamic tracing In broad terms the selection between linear tracing and ring-mode tracing, can be made by deciding whether the problem is cyclic or singleton Cyclic problems (which often encompass performance issues) are often addressed using linear tracing Singleton problems are often addressed with ring-mode tracing

6 Linear mode tracing Linear mode tracing is defined as tracing over a either a controlled period of time, or for a given amount of data Most often, linear mode tracing will make use of either the standard tracelogger or qconn utilities, although a custom utility can also be the appropriate choice Tracelogger is a command-line utility Qconn is a daemon process that allows the Momentics IDE to control the tracing process remotely Qconn allows much finer-grained control of run-time static filtering, than is possible with tracelogger Custom written utilities, can use the TraceEvent() API, to implement either static filtering (i.e. as with qconn) or dynamic tracing/filtering/statistics, none of which are possible with either qconn or tracelogger When linear tracing is required over a relatively long interval, static/dynamic filtering or dynamic tracing will often be necessary in order to maximize the duration which can be recorded in a given amount of storage space

7 Ring-mode tracing Ring-mode tracing is defined as tracing which takes place over an indefinite period, but only results in storage of the trace data when an event of interest occurs Ring-mode almost always involves development of a custom utility (or instrumentation of existing code) to detect the event of interest, and (like linear mode) can involve either static or dynamic run-time filtering as well Ring-mode can be implemented as a two phase process, when there is a known event that precedes the event of interest Phase 1, the user developed code disables tracing, and waits for the pre-cursor event to occur Phase 2, upon detection of the precursor event, tracing is enabled, and the user code waits for the event-of-interest to occur (at which point tracing is stopped, thus producing the trace file This two phase approach can reduce the need for filtering, resulting in the ability to capture more detailed data preceding the event-ofinterest

8 Static filtering Static filtering (at runtime) is defined as filtering based a static mask of events to be traced The tracelogger utility provides only crude control of these static filters The qconn daemon (via the Momentics IDE) provides full control of these static filters (for the duration of the trace) Dynamic tracing can modify these otherwise static filters

9 Dynamic filtering Dynamic filtering is defined as runtime filtering that can change over time based on the state of the system Does not use trace masks like static filtering Filters are established by attaching user event handlers to each event/class type to be filtered Arbitrary user provided C code in this handler determines whether event should be traced, or dropped, and returns either a zero (drop) or a one (trace) As with static filtering, dynamic filtering may be controlled by dynamic tracing

10 Dynamic statistics Dynamic statistics is defined as the runtime collection of event statistics Dynamic statistics are implemented using the same mechanism as dynamic filtering (i.e. the handler callbacks) User supplied code in the event/class handler callback counts events This can be as simple as simply counting each event, or it can be dynamic in that when certain event counters hit certain values, then other statistics counters are enabled, that also record related data An example might be a handler attached to mmap(), that when the number of calls to mmap crosses a threshold over some unit of time, signals a thread to insert a marker user trace (making analysis much easier) Dynamic statistics (like static and dynamic filters) can be controlled by dynamic tracing

11 Dynamic tracing Dynamic tracing is defined as tracing that modifies static/dynamic runtime filters and/or dynamic statistics based on a model of the system applied to the current state of that system This is useful for both chronic analysis (performance) and acute analysis (behavioural defects) An example might be an HA manager process, that focusses tracing on a particular process that it detects is mis-behaving (for instance, it might turn on wide-mode for a class of events related to the undesirable behaviour Dynamic tracing is an extraordinarily powerful technique for instrumenting a fielded system to collect high value traces without user intervention (i.e. it can be a critical component of HA) Dynamic tracing is not something that would typically be employed to analyse a specific defect, since custom code will generally be even more effective than a generalized approach

12 How does it work? The following diagram gives an overview of the mechanism involved in collecting trace data Process/thread creation System calls Interrupts On/Off filters (tracelogger) Static event filters (qconn) Dynamic filters (user code) Events Microkernel State changes E1 E2 E3 E4 E5 E6 Event buffers Visualization (System Profiler) Network Capture File

13 Tracelogger The utility that is most often mentioned when discussing trace acquisition is tracelogger The tracelogger utility serves three purposes It is a useful stand alone utility for linear tracing It is useful when used in conjunction with user supplied code in ringmode tracing Its source is a useful example reference of the TraceEvent() API The tracelogger command line options are described in the QNX documentation suite A simple invocation for linear tracing might be tracelogger w s10.0 This would collect 10 seconds of trace data with ring0 events removed, and all other events collected in wide mode A simple invocation for ring-mode might be tracelogger k256 r This would allocate 256 kernel buffers, and then wait for a _NTO_TRACE_STOP event, at which point it would unload the kernel buffers into a trace file

14 Qconn Qconn (Qnx CONNector) is the daemon that allows the IDE to connect to the target, and control tracing remotely Qconn can be started on cards/blades that don t have a management ethernet port, and (using Qnet) make use of the RP s management ethernet To use qconn on cards/blades that lack an ethernet port of their own, you need to start qconn with the environment variable SOCK set to point to the RP, and to utilize a port that does not conflict with another instance of qconn Example: SOCK=/net/node0_rp0_0 qconn port=8010 Start qconn on a linecard using the network stack on the RP, and listening on port 8010

15 TraceEvent() API The TraceEvent() API is the single most important feature when doing advanced debugging with the System Profiler The TraceEvent() API is document in the manuals Probably the simplest use of the TraceEvent() API that also illustrates the most significant results for debugging is the TraceEvent(_NTO_STOP_TRACE) call Using this single call, in combination with the tracelogger ring mode invocation, yields a backtrace (depth determined by number of kernel buffers and system load) from the detected fault Many DDTS s have been resolved using this technique, and knowledge of how to use this technique is fundamental to basic competency in the System Profiler

16 Analyzing a Trace Rudiments of basic analysis

17 Overview of the System Profiler The System Profiler is the tool that renders the binary trace file It is accessed at Window->Open Perspective->Other: QNX System Profiler The profiler has seven basic editor views, these are: Summary View Partition Summary View CPU Activity View CPU Usage View CPU Migration View Inter CPU Communication View Timeline View These are accessed by using the menu The profiler also has thirteen additional panel views icons, to access the pull-down These views are accessed at Window->Show View->Other: QNX System Profiler

18 Summary View The System Profiler Summary View, provides some basic data about the trace, and is the default view rendered Interesting data includes: Pie chart of CPU allocation between Idle User System Interrupts

19 Partition Summary View The Partition View Summary displays basic data about how partitions are being utilized Interesting data includes: Pie chart of CPU allocation by partition Desired and observed CPU for each partition For each process, the total CPU utilization, and total in each partition

20 CPU Activity View CPU Activity View displays an overview of CPU utilization by core over time

21 CPU Usage View CPU Usage View provides CPU utilization per: Thread Core Priority Partition

22 CPU Migration View CPU Migration View displays the number of times a pre-empted thread is rescheduled on a different core than it was pre-empted from

23 Inter CPU Communication View Inter CPU Communication View displays the number of times a receiving thread unblocks to receive a message from a thread on a different core

24 Timeline View Timeline View is the classical System Profiler View; it provides detailed information regarding a threads state and activities over time, and resembles the display of a hardware logic analyzer Trying to scroll to the solution, via the Timeline View will not be productive The Timeline View should be used when the problem has been isolated using the other editor views, panel views or the trace search facility Views useful for isolating faults are: CPU Usage View (editor view) Partition Summary (editor view) Client Server Statistics (panel view) Why Running? (panel view) Condition Statistics (panel view) Thread Call Stack (panel view)

25 Searching Using the trace search facility

26 What is the Trace Search Facility? The Trace Search Facility is one of the most often used features of the System Profiler when analyzing a trace and, as such, deserves its own section The Trace Search Facility is accessed via Search->Search: Trace Search The Trace Search dialog provides the ability to define a combinatorial search condition, for any of the events in the trace (including user defined events) The Trace Search dialog will pre-scan the trace file, and present only events that actually exist in the trace (so you don t waste time searching for something that doesn t exist at all) The Trace Search dialog also allows parametric searches for each individual event

27 Using the Trace Search Dialog As a simple example, we will configure a search that locates all sends that qconn does to proc, and all receives that proc does on messages from qconn First step is to open the Trace Search dialog

28 Using the Trace Search Dialog (cont) Next, let s configure a condition to detect a send from qconn to proc Click Add in the Defined Conditions group, you ll then see the following Give the condition a name (MsgSend sounds good here) Select a communication class event Select a Send Message event code

29 Using the Trace Search Dialog (cont) Next, select just the element within which to search for the condition In this case, that is qconn, since we want to find the send side of the message So first, uncheck the select all elements checkbox, then select qconn from the list of searchable elements Restrict to pid 1 (proc) Uncheck Select qconn

30 Using the Trace Search Dialog (cont) Now click Finish, to close the condition definition dialog

31 Using the Trace Search Dialog (cont) Next, repeat the steps to define a condition, but this time define a condition as follows Name it Choose a kernel call Select MsgReceivev Exit Restrict search to proc Enter the pid of qconn

32 Using the Trace Search Dialog (cont) Now click Finish, to close the condition definition dialog

33 Using the Trace Search Dialog (cont) Next, set an or ( ) conditional between the MsgSend and MsgReceive conditions, when done click Search

34 Using the Trace Search Dialog (cont) You should see something like this in the Timeline and Search Views

35 Utilizing the Search Results in Condition Statistics Now you can utilize these search conditions to configure a Condition Statistics View First bring the Condition Statistics View to the front Next, click on Configure Table Note that in the upper right there is an icon that allows you to restrict the condition statistics to a selectable range

36 Utilizing the Search Results in Condition Statistics (cont) After clicking on Configure Table you should see the following dialog Select both MsgSend and MsgReceive and then click OK

37 Utilizing the Search Results in Condition Statistics (cont) After clicking on OK, a dialog will prompt you to rerun the statistics, click Yes, and then you should see something like this This shows the number of MsgSends (by qconn) and the corresponding number of MsgReceives (by proc) Of course, this is just a trivial example, but the condition statistics (which are really part of searching) are incredibly valuable

38 Case Studies Reviewing some case studies of actual DDTS s