CorreLog. Pivot Report Generation Function Application Notes and User Guide

Transcription

1 CorreLog Pivot Report Generation Function Application Notes and User Guide

2 CorreLog, Pivot Report Generator Application Notes Copyright , CorreLog, Inc. All rights reserved. No part of this manual shall be reproduced without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibilities for errors or omissions. Nor is any liability assumed for damages resulting from the use of this information contained herein. Pivot Report Application Notes, Page - 2

3 Table of Contents Section 1: Introduction.. 5 Section 2: Pivot Report Usage.. 7 Appendix A: Configuration Tutorial.. 28 Appendix B: Pivot Test Vectors.. 33 For Additional Help.. 35 Pivot Report Application Notes, Page - 3

4 Pivot Report Application Notes, Page - 4

5 Section 1: Introduction This application note provides a detailed discussion and overview of operation for the CorreLog "Pivot Report" facility. This is a standard function of all CorreLog servers, and allows the operator to analyze certain types of log data. The Pivot Reporting function can be thought of as a general-purpose log file analyzer that can assist a operator with inspecting the structured data of messages, possibly as part of forensics associated with a security event. The "Pivot Reports" tool is used on message threads that contain specific fields and structured of data. These types of messages include firewall log data, HTTP server log data, and many other message types that contain "field based" data. All the messages in a particular CorreLog thread are parsed, and the values of specific fields (called "pivot items") are extracted and tabulated. The operator can then see the unique values for various message fields spanning large numbers of messages, such as all the status codes, URLs, device names, source and destination addresses, and other data contained in the log data. Brief information on Pivot reports is presented in the "CorreLog User Reference Manual" and "CorreLog Screen Reference Manual", both of which are available from the "Home" screen of the CorreLog Server. The application note herein furnishes information not otherwise covered in those manuals, including a description of all screen components and features of the Pivot Report configuration and processes, as well as the database interface to the Pivot Report facility, which allows report data to be entered into a operator specified database table for analysis by third-party report generators. Pivot Report Application Notes, Page - 5

6 General Description The operator accesses Pivot reports via the "Reports > Pivot" screen. The toplevel screen shows the various Pivot reports that have been configured. These reports are generated at midnight (like other reports) and can also be generated on demand. Beneath each report title are a series of links that define the "pivot items" that have been configured by the operator. The operator clicks on any of these links to see the data associated with the pivot item, including a graph of the occurrences for the pivot item. From that screen, the operator can further drill down to see the precise message data associated with the pivot item. Initially, no Pivot Reports are defined. The operator must define one or more pivot reports by clicking the "Add New" button at the top of the screen, which allows the operator to specify a thread to operate upon, a report title, and additional match patterns. As part of the Pivot Report setup, the operator must also define the fields associated with the Pivot report by clicking the "Config" button. The Pivot Report contains several advanced features, accessed via the "Advanced" button on the top-level screen. These advanced features include the ability to load pivot data into a relational database, for use with third-party reporting tools. Specific advanced features are discussed at the end of this manual. Definition Of Pivot Items CorreLog refers to "Pivot Items" as the operator assigned names of specific fields that may occur in the message. For example, for a particular type of message, the fourth word of a message may contain a source IP address. In this case, the operator could label the "Pivot Item" with an arbitrary name such as "SrcIP", and the report generator will discover all the possible values for this field (to a operator configured limit) and depict a bar chart for all Source IP addresses on the system. Each message class can have multiple pivot items defined for that class. For example, an HTTP Server Log may have "URL", "Address", "Browser", "HTTP Status", and other pivot items of interest. Similarly, a firewall may have "SrcAddr", "SrcPort", "DestAddr", and "DestPrt", type pivot items. The operator configures these items interactively, by sampling the message, inspecting the structure of the message, and then entering a value next to specific fields of interest. When the report generator runs, the operator can drill down from the top-level screens to see the various pivot item values in bar chart format. Pivot Report Application Notes, Page - 6

7 Section 2: Pivot Report Usage The CorreLog Pivot Report is an embedded function of all CorreLog Servers, and does not require any plug-ins or adapters to fully access and use. The facility is accessed from the "Reports > Pivot" tab of CorreLog, which provides a series of screens that permit configuration and viewing of the Pivot data. The CorreLog "Reports" application group, where the Pivot Reporting facility resides, provides general utility in the reporting of both raw and correlated message information. In addition to the "Pivot Report" facility discussed here, these other reporting screens include a graphing facility, as well as a comprehensive reporting facility based on Microsoft Excel spreadsheets. The report generators normally run at midnight each day, collecting information from the previous day, and making this data available to operators the next morning. Reports can also be generated on demand, such as to view the latest Pivot data for the current time. Pivot reports operate in a similar fashion to Excel reports (documented extensively in the "CorreLog User Manual") except that their setup and operation is slightly more complicated. Unlike Excel reports, which require very little configuration other than selecting a "Thread" and "Report Title", the Pivot facility requires the operator to define the structure of the data, and the actual "Pivot Items" that are reported upon. This activity is simplified by a user-friendly configuration facility. Both the top-level reporting and configuration screens are documented in this section. Pivot Report Application Notes, Page - 7

8 Top-Level Pivot Report Screen A typical top-level Pivot Report screen, accessed via the "Reports > Pivot" tab, is depicted below. This picture shows examples of several Pivot reports. (These reports are not included with CorreLog; the operator must define one or more reports as discussed in Section 3 of this manual.) The above screen depicts a typical top-level screen with three different Pivot Reports configured. The actual look of the screen depends upon the types of reports configured, and will vary between systems. Note that the Pivot Report screen will initially display a message indicating that no reports are currently configured. (The operator must click the "Add New" button to add one or more pivot reports to the system. Specific elements of this screen are as follows. Refresh Button. The "Refresh" button performs the standard CorreLog function of refreshing the screen to see the latest changes on the system. Pivot Report Application Notes, Page - 8

9 Add New Button. The "AddNew" button performs the standard CorreLog function of allowing the operator to add a new pivot report. The button displays the "Edit" screen documented in the next section. Run Report Button. The "Run Report" or "Generate" button allows the operator to regenerate all the reports on the screen. (These reports are automatically generated at midnight.) The operator can also drill down into a particular report (via the "Edit" button) and regenerate the single report. Advanced Button. The "Advanced" button allows the operator to access the advanced parameter settings for the Pivot Report generation process, which includes the ability to load data into a database, as well as other functions. Status Message. Beneath the buttons, a status message appears indicating the current status of the report generator. The value of the status message depends upon several factors, such as whether the report generator is currently running, whether errors have been encountered, etc. Report Table. Beneath the status message appears the report table, providing the listing of all reports currently configured on the system, as discussed below. Edit #NN Buttons. Within the "Report Table", the first column is an "Edit" button, which allows the operator to modify (or delete) the associated pivot report. This button performs the standard CorreLog function of other "Edit" buttons within CorreLog. Pivot Report Name. Within the "Report Table", the second column is the name of a Pivot report, selected by the operator. This name describes the contents of the report, and is arbitrary text to assist with identifying the report. Pivot Item Links. As part of the Pivot Report Name (above), a series of links to each pivot item allows the operator to access a particular pivot report. These values correspond to the labels associated with the pivot items, configured by the operator on the "Configure" screen. Time Last Accessed. The rightmost column of the Report Table contains the time that the report was last accessed by an operator; useful for determining which Pivot Reports have been recently viewed. Pivot Report Application Notes, Page - 9

10 Pivot Report Edit Screen The operator adds or edits a pivot report using the "Pivot Report Edit" screen. This screen is accessed by clicking on the "AddNew" button at the top of the screen (to add a new report) or is accessed by clicking the "Edit" button to the right of an existing report. The "Edit" screen is depicted below. The "Edit" screen is a standard CorreLog dialog with "Cancel", "Reset", and "Save' buttons. The actual screen varies slightly, depending upon whether the operator has accessed the screen via the "AddNew" button, or via the "Edit" button (noted below.) Specific elements of this screen are as follows. Cancel Button. The "Cancel" button performs the standard CorreLog function of canceling the edit operation, and returning the operator to the top-level Pivot Report screen. Reset Button. The "Reset" button performs the standard CorreLog function of discarding any changes and redisplaying the initial information Pivot Report Application Notes, Page - 10

11 (identical to clicking the "Cancel" button and then reselecting the "Edit" report. Delete Button. (Edit Screen Only.) The "Delete" button performs the standard CorreLog function of deleting all information associated with the Pivot report. This button appears only on the "Edit" screen, and not the "AddNew" screen. Save Button. The "Save" button performs the standard CorreLog function of saving the information and returning the operator to the top-level Pivot Report screen. Save & Configure Button (Add New Screen Only.) The "Save & Configure" button appears ONLY on the "Add New" screen, and saves the current report and then immediately goes to the "Configure" screen. (The "Configure" screen can also be accessed on both the "Add New" and "Edit" screens using the "Configure" button towards the middle of the page, as described below. Run Report Button. (Edit Screen Only.) The "Run Report" or "Generate" button queries the operator to generate the selected screen, providing a method of generating only one Pivot report. This can save the operator time when testing a report configuration, or when the operator needs only a single report. Note that this button discards all other Pivot Report data for the day, including data associated with other Pivot reports. Report Data Source. This drop-down menu lists the source of the data for the pivot report. The operator selects any existing "Thread", or any "Aux" file on the system. The pivot report will use that particular data source for all the message information used to create the report. Pin To Top. (Edit Screen Only.) This select menu performs the standard CorreLog function of pinning the report to the top of the list, as part of the user preferences. This option appears only on the "Edit" screen. New reports are always pinned to the top for the user that created the report. Report Title. This text input field is the report title, which appears on the top-level screen. The title should describe the nature of the report, and can be modified after the report is created. The title can use a wide character set. Pivot Parse Mode. This select menu specifies the parse mode for the report. Two different parse modes are available: the default "Use Field Specs" allows the operator to select fields based upon field position, and is the default mode. The "Use Parse Specs" mode allows the operator to select fields based upon CorreLog parse specifications. This setting Pivot Report Application Notes, Page - 11

12 affects the operation of the "Config" button. (See next section for a detailed description of the "Parse Mode" setting.) Pivot Field Specification. This text area includes the pivot specification, and is displayed only if the "Pivot Parse Mode" (above) is set to "Use Field Specs". Although this area can be hand edited (such as to change a label field) it is best edited via the "Config" button below. Configure Button. This button launches the "Pivot Item Configuration" screen, which allows the operator to assign labels, specify delimiters, and perform other parsing functions that may be necessary to identify the various fields as pivot items. (See next section.) The actual operation of the button depends upon the "Pivot Parse Mode" select menu above. The operator will generally click this button at least once when configuring the pivot report. Report Span Days. This select menu specifies how many days the report will span, by default one day. Selecting a high value for this item may greatly increase the time to generate a pivot report. When the "Max Data Records" value is reached, or when the "Report Span Days" value is reached, the pivot report stops further processing of messages. Max Data Records. This select menu specifies the maximum number of records to process. When the "Max Data Records" value is reached, or when the "Report Span Days" value is reached, the pivot report stops further processing of messages. The value can be used to limit the scope of the report and time to generate the report. ODBC Data Source Name. This select menu allows the operator to specify an ODBC data source. The value is optional. (See "Advanced Configuration" section, below.) If the operator selects a value other than "None", the operator must also specify a database table name, which will receive the pivot report data. This value requires the operator to set the "Enable Pivot Report Output" value (on the "Advanced Configuration" screen) to be "True", or the setting has no effect. Database Table Name. This text area specifies the name of the database table, valid only if an ODBC Data Source Name is specified. (See above.) If an ODBC data source is specified, the operator must also specify a database table name, which will receive the pivot report data. Pivot Report Application Notes, Page - 12

13 Pivot Item Configuration Screen As part of creating a new pivot report, the operator must define the fields within messages that will be parsed. This is accomplished by first clicking the "AddNew" or "Edit" button (to access the Pivot Report Editor screen) and then clicking the "Configure" button to configure the field and label associations. This displays the "Pivot Item Configuration Screen". Two different modes of operation exist for the Pivot Item Configuration Screen, selected via the "Pivot Parse Mode" menu on the top-level edit screen. A description of these modes is as follows: Use Parse Specs. This is the default setting, and is the most convenient way to configure the Pivot items. The operator is permitted up to eight different parse specifications, where each parse specification consists of a field number, a keyword prefix for the data, or a parser function. This type of parse function is especially useful for any type of data where certain prefix codes or more complex parsing is necessary, such as Windows messages, UNIX style messages, or application data that is not highly structured. Use Field Specs. This is a wholly alternate setting which is more difficult to use than the default setting above, but may offer some advantages to configuring highly regular data where each message item (delimited by some value) is always in a fixed location. This setting allows the operator to delimit fields by certain characters and qualifiers, and then assign a label for each field. The type of "Pivot Parser Mode" setting selected by the operator will depend upon the type of messages to report on. For most applications the default "User Parse Specs" will be adequate, and will almost always be easier to use than the more complicated "Use Field Specs" setting. (For this reason, the "Use Field Spec" setting, while still supported, is largely deprecated.) Each of the above two settings are described in detail within the sections that follow. Each parse mode has its own special features and functions suitable for use with a wide variety of different message formats commonly found in SIEM systems. Pivot Report Application Notes, Page - 13

14 Pivot Item Configuration Using "Parse Specs" The default "Use Parse Specs" setting, discussed in the previous section, permits the user to specify the pivot items using the standard CorreLog parse functions. If the operator has selected the default "Use Parse Spec" parse mode, then clicking on the "Configure" button accesses a screen containing a series of eight different parse specifications. The operator may provide arbitrary parse specifications and label each field. The "Configure" screen, when the "Use Parse Specs" value is set, is depicted below: On the above screen, the user provides a standard CorreLog parse specification, and provides an arbitrary label for the field. The user can select a "Common Field" (parsed from the message) via drop down select menus, and the parse specification will be inserted into the "Parse Spec" field. The operator can also enter a parse specification directly.. Brief help on parse specifications is provided in the "Parse Help" link. A parse specification can be a field number (such as 1,2,3, etc.) or can be a match Pivot Report Application Notes, Page - 14

15 pattern such as "user name: *" or "account name: *", or can be a more complex parse function such as $integer(fieldno). Detailed help on parse specifications is available in a variety of locations, including the "CorreLog User Reference Manual". In addition to standard "Cancel", "Prev", "Refresh", and "Continue" buttons at the top of the field, this "Configure" screen provides the following inputs and controls: Display Sample Messages. At the top of the screen the user can click the "Display Sample Messages" hyperlink to display two sample messages that may assist with the configuration process. The sample messages assist with creating parse specifications based upon the current data within the catalog. Parse Spec. Eight different input fields are provided that allow the operator to configure eight different parse specifications. Each parse specification can be a field number, a match pattern, or a parse function. Clicking on the "Parse Help" link provides brief help on using parse specifications in the system. Field Name. If the operator specifies a Parse Spec for one of the eight different slots, the operator should provide a label for the parse spec. This label appears on the top-level "Pivot Report" screen, and can be clicked to view the messages associated with the parse specification. If the user does not specify both a "Parse Spec" and "Field Name" value, the link does not appear on the top-level screen and this particular slot is skipped during report generation. Insert Common Field. This select menu contains common fields that are parsed from the list of recent messages. The user selects the common field and its parse function and label are automatically inserted into the "Parse Spec" and "Field Name" inputs, replacing any existing value. These common fields are representative of the particular messages within the catalog, and may or may not include all common fields of interest. Additional Parse Specs. Beneath the Parse Spec, the operator can click the "Additional Parse Specs" link to open up two additional parse specifications that are used (in sequence) if the first parse spec fails to match a portion of the message. If the main parse spec fails to match a portion of the message, the report generator will try the second (and then the third) parse specification. This permits the operator to easily configure a label that works with two different message types. This is commonly used in parsing Windows 2003 and Windows 2008 messages (which have different parse specifications for the same general item, such as "user Pivot Report Application Notes, Page - 15

16 name: *" for one specification and "account name: *" for an alternate specification. Match Delimiter. At the bottom of the screen, the "Match Delimiter" value permits the operator to specify a delimiter for the parse specifications. The values are "space", "semi", "comma", "colon", "pipe", "sp-dash", "dash" and "punct". A value of "sp-dash" delimits the parsing action by a space, followed by a dash (hyphen) character. A value of "punct" delimits the parsing action by any punctuation mark. The selected delimiter ends the parse function. The value can be selected by inspecting the sample messages. Most messages are easily parsed using the "space" delimiter (the default), hence this setting is mainly useful for more complex parsing operations associated with some messages. Pivot Report Application Notes, Page - 16

17 Pivot Item Configuration Using "Field Specs" As an alternative to using "Parse Specs", the operator can select the "Use Field Spec" parse, then clicking on the "Configure" button to view an alternate configuration screen. This screen displays two sample messages, and allows the user to provide a label next to any field of interest. The "Config" screen, when the "Use Field Specs" value is set, is depicted below: The "Pivot Item Configuration Screen" is a moderately complex screen with several different parts. The operator can click the "Delimiter and Normalization Options" tab to view the special word and phrase delimiters. The operator can also click the "Pre-processing Options" hyperlink to display specific preformatting options. Both of these special features are discussed in the next section. The main elements of the screen are discussed below. Prev Button. The "Prev" button returns the operator to the previous "Edit" screen without further action. Pivot Report Application Notes, Page - 17

18 Apply Button. The "Apply" button applies the delimiters and filter patterns, selects new sample messages using these values, and refreshes the screen. Note that the "Apply" button clears any field definitions that the operator may have constructed. Continue Button. The "Continue" button saves the settings of the screen, and returns the operator to the previous "Edit" screen. Delimiter Options Hyperlink. This link, when clicked, displays delimiter options that determine how the sample messages are parsed. The operator can select various delimiters, patterns and pre-format options. (A complete discussion of these functions is provided in the next section.) Sample Messages #1 and #2. The main part of the display consists of a table with three columns: (1) The first column contains a sample message broken down into words and phrases, as set by the "Delimiter Options" hyperlink; (2) The second column contains a second sample message, for reference; (3) The third column contains the label to be applied to the particular field. The operator can select new sample messages via the "Select Different Samples" button at the bottom of the screen. This table defines the labels for message fields, which appear on the top-level screen. The exact method of selecting a field depends upon the operator inspection of the message, and several techniques apply. (See section on "Selecting Fields and Sub-Fields Within The Message".) Field Names. Next to the two sample messages, at the right, are the operator specified field names. For each field of the sample messages, the operator can enter a value that will appear on the top-level "Pivot Report" screen. (See below.) Select Different Samples Button. "The "Select" button, appearing at the bottom of the screen, discards the two sample messages and displays the next two sample messages (if any) that appear for the thread. This allows the operator to quickly see the nature and type of messages that will be parsed by the report. Note that the "Select" button clears any field definitions that the operator may have constructed. Pivot Report Application Notes, Page - 18

19 Pivot Item "Field Specs" Values On the "Pivot Report Configuration Screen" the operator simply adds arbitrary text labels to identify the field content. These text labels are short keywords, such as "URL", "FromDest", "MsgStatus", "ErrCode", etc. The user selects the name of the label, which then appears on the top-level Pivot report screen to identify the particular field of the message. If a label is not added to a field, the particular field is ignored during the generation of the pivot report. Not all fields will normally be identified; only those fields of particular interest will have a label. Also, if a label is assigned to a field that is constantly changing (such as a timestamp) the performance of the report generation may be slightly degraded, and the information acquired will probably not be pertinent to the operator, since there will be no common pivot items values among the messages. Special Field Spec Names And Label Prefixes There are several "special" label prefixes, which the operator can use to perform special functions. Each of these special prefix values begins with a "$" character, and the functions correspond loosely to the "Parse Function" names found elsewhere in the system. The prefixes permit the user to perform special processing on a field, such as match a particular field type, and ignore fields that do not have a special format (such as ignoring fields that are not IP Addresses, or URLs.) These prefixes are not case sensitive; the operator specifies a field name such as "$ipaddr", or "$IPAdd.Src" or "$IPaddr_Dest" when assigning the field label, which will format the field as an IP address, and bypass messages that do not contain an IP Address in the selected field position. Specifically, the special field prefixes modify the operation of the report generator as follows: NOTE: These special naming conventions apply ONLY to the "Use Field Specs mode of operation, and have no special meeting when using the default "Use Parse Specs" mode, discussed earlier. $Geo Prefix. A prefix of "$Geo" can be used if the field contains an IP address. This causes the IP address to be parsed from the field, and the Country Code for the IP address substituted for the Pivot label. (A list of Country Codes can be found on the "More > Geo-IP Tool"). This is especially useful when analyzing firewall messages that contain external IP addresses. For example, if the fifth word of a message is the source IP address, the operator can specify a label such as $Geo_Source" to categorize the data by country of origin. Pivot Report Application Notes, Page - 19

20 $URL Prefix. A prefix of "$URL" can be used if the field contains a URL. This causes the URL (beginning with " or " to be formatted. If the field does not contain an URL, the message is skipped. This is mainly useful as a method of filtering out irregular messages where a particular may not always contain a URL as its value. $ Prefix. A prefix of "$ " can be used if the field contains an e- mail address in the form X@Y.Z. The field is formatted as an address. If the field does not contain an address, the message is skipped. This is mainly useful as a method of filtering out irregular messages where a particular may not always contain an address as its value. $Integer Prefix. A prefix of "$Integer" can be used if the field contains a numeric value. Any alphabetical characters are stripped from the field, leaving an integer number value. If the field does not contain any numeric content, the message is skipped. This is mainly useful as a method of filtering out irregular messages where a particular may not always contain a number address as its value. $IPAdd Prefix. A prefix of "$IPAdd" can be used if the field contains an IP address. This operates in a fashion similar to the $Geo prefix, and causes the IP addressed to be parsed from the field. This is useful if the field contains other text, in addition to a regular IP address. If the field does not contain any IP address, the message is skipped. $Basename Prefix. A prefix of "$Basename" can be used if the field contains punctuation marks. Only the information preceding the first punctuation mark is returned as the pivot item. For example, a field containing the value "README.txt", returns a pivot item value of "README". $Suffix Prefix. A prefix of "$Suffix" can be used if the field contains any punctuation marks. Only the information following the last punctuation character is returned as the pivot item. For example, a field containing the value " :80" returns a pivot item value of "80". Note that the above prefixes perform two different functions: (1) First, the prefix can be used to format a field's value, such as the $IPAdd prefix (which will return a value of " " for a field "Src= :99"). This first function can be used to "Cast" a field value into a particular type. (2) Secondly, the prefix can be used to bypass messages where the field value may be irregular, for example skipping message processing if a certain field in a message is not numeric, an address, etc. Pivot Report Application Notes, Page - 20

21 Field Spec Delimiter and Normalization Options Clicking on the "Delimiter and Normalization Options" hyperlink at the top of the "Pivot Item Configuration Screen" (discussed in the preceding section) opens up a new table of special options that determine how the sample messages are parsed. The operator can specify special delimiters that affect the fields of the sample messages, and can perform special "Pre-Processing" on each message to modify the message format before it is parsed. These options control how the class of messages is parsed into separate fields. By default, the system parses words at spaces, and phrases at double-quote marks. The operator can select other parsing options via checkboxes. The system distinguishes between words and phrases. Specifically, a phrase can have multiple field definitions. A phrase may contain multiple words, and the operator can configure the parser to select any particular word within the phrase (where words are delimited by space, comma, semi-colon or colon, or any combination of these.) By default, the entire message is considered one phrase. However, the operator can break the message up into several phrases, and operate on each phrase separately. This provides simplicity in the general case, as well as considerable power in parsing messages that may contain multiple phrases. A discussion of this feature is provided in later sections. Word Delimiters. The operator can check of the word delimiters, either "space", "comma", "semi-colon, or "colon." The message is broken up into separate fields based upon the word delimiters. Phrase Delimiters. Phrase delimiters are a special case of delimiters, which break the message up into phrases based upon either "double quote", "bracket", "parenthesis", or "brace", characters. Each phrase can then be further parse by word delimiters. (See section below.) Additional Match Expression. The operator can inspect the match expression (input on the previous "Edit" screen) and make adjustments to filter out messages that may be in the selected input thread which do not match a particular structure. This permits the operator more flexibility in selecting messages of a particular type or class, necessary to consistently parse the message into fields. Note that this is the same "Match Expression" available on the parent "Edit" screen, included on the "Config" screen for convenience. Pre-Processing Options. The operator can click on the "Pre-processing Options" hyperlink to apply a global match and replace pattern to each message immediately before it is parsed by the "Word" and "Phrase" delimiters. This screen allows the operator to specify a "from pattern" and "to pattern", and specify how the "from pattern" is matched, either "First", "Last", or "All" occurrences of the "from pattern." This allows the operator Pivot Report Application Notes, Page - 21

22 to insert delimiter characters in the message prior to any parsing, which may be required for certain highly complex messages. Selecting Phrases And Sub-Fields Within The Message In the special case where a message is composed of several different phrases (delimited by "double quote", "bracket", "parenthesis", or "brace" characters) the operator can configure multiple fields for the phrase, to match one or more subfields within the phrase. For example, consider a group of message such as the following: Field1=ValueA Field2="ValueB ValueC ValueD" Field3=ValueE Field1=ValueB Field2="ValueC ValueD ValueE" Field3=ValueF Field1=ValueC Field2="ValueB ValueE ValueF" Field3=ValueG The contents of Field2 represent a specific phrase, delimited by double-quote marks. The operator can parse any or all of the fields of this phrase by entering a field description such as the following for these values: "* * Substring" will associate the label "Substring" with the third word of the second phrase, and this will be the pivot item. Phrases are depicted with parenthesis above and below the field label, indicating to the operator that the phrase can be further specified using the syntactical convention described above, where "*" skips a label for the word within the phrase, and any other text causes the word within the phrase to be a pivot item. This technique is rather advanced, but may be necessary for a wide class of messages (including Apache Log Files) to select certain precise pivot items from the message. Pivot Report Application Notes, Page - 22

23 Contrasting "Parse Specs" and "Field Specs" Modes As discussed in the preceding sections, the two alternate parse modes (selectable from the top-level edit screen) provide the same purpose and intent, which is to record the unique values of a message sot that counts and related messages can be labeled. Except for the fact that the data is parsed in different manners, both the "Field Spec" and "Parse Spec" modes create the same report. (The top-level "Parse Mode" setting affects the way the data is parsed from the message database, whereas the actual end report is the same for both cases.) Prior to Version 5.2.0, the "Field Spec" parsing method was the only way to parse data. In current versions, the "Parse Spec" mode of operation (which is the default) will generally be easier and more flexible to implement. The "Field Spec" method provides a simple way of creating a report where the data is positional, for example where the fifth word is usually or always an IP address, the sixth word is usually or always a numeric status code, etc. This applies to a wide variety of structured messages, but not all messages. The default "Use Parse Spec" technique expands the role of the Pivot report to include irregular data, free form text, where fields are identified by regular prefixes, suffixes, or parse functions. Additionally, unlike the "Use Field Spec" setting, the "Use Parse Spec" setting allows the operator to process messages that have completely different positional formats and keywords (such as aggregating "Windows 2003" and "Windows 2008" message types.) Finally, note that the "Parse Mode" setting is set on the "AddNew" screen, and cannot be changed after the report is created. (The user can view, but not change the "Parse Mode" setting on the "Edit" screen.) To change the parse mode, the operator should simply delete the existing report and create a new report. This will not be an impediment to the operator, since configuration of a Pivot report is a fairly simple operation. Pivot Report Application Notes, Page - 23

24 Advanced Pivot Report Parameters Clicking on the "Advanced" button at the top right of the top-level Pivot Report screen displays a dialog of advanced options that allow the operator to specify operational parameters for the system. These parameters may or may not be required to generate or operate Pivot reports, depending upon the operator's particular applications. A depiction of this screen is shown below: The above screen is a standard CorreLog dialog that allows the operator to modify parameters that effect the execution of the report generator. Specific values are as follows: Enable Pivot Catalog Output. This select menu enables the generation of catalogs for each pivot item (i.e. the messages for each pivot item are collected and viewable.) This value is normally set to "Yes". Setting this value to "No" will disable the collection and cataloging of messages, but will still result in the generation of the pivot report counts and bar chart. Setting this value to "No" can increase the speed of the report generation. Pivot Report Application Notes, Page - 24

25 Enable Pivot Summary Report. This select menu enables the generation of a summary report, permitting the summary report to be ed, or downloaded in HTML or PDF format. This item is normally set to "No". If set to "Yes", then Pivot reports will appear in the "Reports > " dropdown list, permitting pivot reports to be ed to end users. Enable Pivot ODBC Output. This select menu enables the output of the pivot report counts to a relational database (where the specific ODBC data source name and table name are specified on the "Pivot Report Edit" screen.) This value is normally set to "No". When set to "Yes", the columns of the table are the names of the pivot items specified by the operator, prefixed with a unique "Database Column Prefix" value (below.) This provides a method of accessing the pivot data from third-party report generators. The database table is dropped and recreated each time the Pivot Report is generated. Require Columns for ODBC Output. This select menu is applicable ONLY if the "Enable Pivot ODBC Output" value is set to "Yes", and is used to limit data entry into the database if only some of the fields in the parse specification were actually matched. The default "Require-All" means that all the fields in the parse spec MUST be matched. In contrast, the "Require-2" setting means that only two fields need to be matched for the message data to be inserted into the database (and the other columns will be NULL values.) This setting can have important implications in the amount of database space necessary to contain a report; the "Require-All" guarantees that there are no records inserted with NULL holes in the table, which may or may not be what the operator wants and needs. Max Unique Values Per Field. This input field specifies the number of unique pivot report items per field (i.e. the number of URLs, Source Addresses, and other items for each label.) The default value is "500", and the maximum value is "2500". This prevents situations where the operator has accidentally selected a field that always changes (such as a timestamp), where there exists an endless variety of unique values for each Pivot Item rather than a fixed number of items that often repeat. Setting this value to its maximum of "2500" will increase the processing required to generate the report, and may result in slow rendering of the report by the operator's browser. Max Database Column Width. This input field is enabled only when the "Enable Pivot ODBC Output" value is set to "Yes", and sets the maximum width of the columns for the database table. The default value is 50 characters, which should be adequate for most situations. The value applies to every column of every table created by the Pivot report. Setting this value too low will cause pivot item values to be truncated, and setting this value too high is wasteful of database resources. Pivot Report Application Notes, Page - 25

26 Database Column Prefix. This input field is enabled only when the "Enable Pivot ODBC Output" value is set to "Yes", and applies a textual prefix to database columns. This provides more flexibility in labeling the columns of the database (which might conflict with the naming rules of the particular database. When enabling the ODBC output, a database table is generated for the Pivot report when the following preconditions exist: (1) The operator has selected a valid ODBC data source name on the "Pivot Report Edit" screen; (2) the operator as entered a valid database table name on the "Pivot Report Edit" screen; (3) the operator has enabled the ODBC output on the Advanced Pivot Report Output" screen; and (4) the values of "Max Database Column Width" and "Database Column Prefix" are valid. Pivot Report Application Notes, Page - 26

27 Appendix A: Configuration Tutorial This section provides specific example procedures and examples of how to use the Pivot Report facility to capture data, including a discussion of the specific steps to populate the system with data, create a thread to collect the data, and then generate reports. The procedure herein relies upon generated test vector messages, which are used to populate the CorreLog "Messages > Search" screen with data. These test vectors are generated by executing a batch file to send structured HTTP data messages (of the type typically generated by Apache Web Servers and other programs) to the CorreLog system using the "sendlog.exe" program. The exact batch file is found in the Appendix to the manual, along with more detailed instructions on how to create and execute the batch file using standard Windows facilities. Once data is generated via the above batch file, the operator creates a thread to contain these messages, re-generates the thread to populate the thread with data, and then creates Pivot reports on the data. This creates realistic Pivot Reports of the type that will be created using actual system HTTP server, Firewall, VPN, or other structured data. The exact steps necessary to accomplish this end-to-end activity are provided in this section, along with a narrative that explains the process in detail. This section can be used by new CorreLog users, as well as developers and quality assurance personnel seeking to test the functionality and operation of the Pivot Report Generation facility. Pivot Report Application Notes, Page - 27

28 Step #1 Populating The Pivot Data The Pivot Report facility operates on a specific type of message consisting of well-defined regular fields, of the type commonly generated by HTTP servers, VPNs, Firewalls, and many other devices. The user can generate this type of data (for the purposes of this tutorial) using the information in the Appendix, as follows: 1. Copy the test vector batch file from the table of the Appendix to a file using notepad. (These test vectors consist of HTTP server messages of the type typically generated by Apache servers, and other HTTP servers.) 2. Copy the above file to the CorreLog\system\PIVOT_TEST.bat file on the CorreLog Server. (This location is required, since the batch file uses the CorreLog "sendlog.exe" utility, found in the system directory.) 3. At a command prompt, execute the above batch file, and verify that the file executes to completion with no errors. 4. At the CorreLog "Messages > Search" screen, verify that the system has been populated with the HTTP server messages within the batch file. Step #2 Creating A Thread Containing the Pivot Data The Pivot report operates on CorreLog Threads (or Auxiliary files), so once CorreLog has received the data, it must be threaded. A correlation thread collects all messages (of a specific type) after it is created. A thread can also be created and "re-generated" to collect past messages. In this particular case, the messages have already been received by CorreLog (in step #1 above), so the operator must define a thread and re-generate the thread as follows: 1. On the CorreLog "Correlation > Threads" screen, create a new thread using the "AddNew" button. 2. Specify a thread title of "Pivot Test Messages", and specify a match pattern of " (which is a keyword of all the messages send in Step #1.) 3. Save the above data to create the thread, and re-access the "Pivot Test Messages" catalog by clicking on the thread name hyperlink. This displays the "Thread Messages" catalog screen. The list of messages will initially be empty. 4. Click on the "Regenerate Catalog Information" hyperlink at the bottom right of the "Thread Messages" catalog screen. This displays the "Regenerate Thread Catalog" screen. Pivot Report Application Notes, Page - 28

29 5. On the "Regenerate Thread Catalog" screen, click the "Generate" button and confirm. This launches the re-generation process and returns the operator to the "Regenerate Thread Catalog" screen. The status of the background process is displayed at the top of this screen. 6. Wait for the catalog to regenerate. When the catalog is finished click the "Cancel" button at the top of that screen to re-access the list of messages. The "Thread Messages" catalog will now contain the messages sent in step #1 of the procedure. Note that regenerating the catalog of messages, as described above, is necessary only in this particular tutorial, to create a thread containing messages that have been already sent. Future messages that match the operator specified keyword will be entered into the catalog automatically, and the catalog will not have to be regenerated. Step #3 Creating the Pivot Report Once the operator has created a thread containing the Pivot Test Messages, the operator can then configure a Pivot report to parse and display this data. The necessary steps to configure this report are as follows: 1. Access the "Reports > Pivot" screen, and click the "AddNew" button to create a new Pivot Report. This displays the "Pivot Report Edit" screen depicted in Section 2 of this manual. 2. On the "Pivot Report Edit" screen, select as the "Pivot Report Data Source" the "Thread/Pivot Test Messages" catalog, populated in the previous steps. (The "Pivot Report Data Source" select menu lists all the threads on the system, any of which can be used as a pivot data source. The user can also select one of the Aux files of the system as a pivot data source, as listed in the select menu.) 3. On the "Pivot Report Edit" screen, provide an arbitrary title for the report, such as "Pivot Test Report." 4. Click "Save" to create the report. This returns the operator to the top-level "Reports > Pivot" screen. The operator can leave the default value for "Additional Match Expression", "Span-Days", "Max Records" and other input items. (The default values for these other items will be suitable for use in generating the example report.) The title specified in step 3 above will appear on the top-level Pivot screen. The link beneath this title will indicate "No data available." Pivot Report Application Notes, Page - 29

30 Step #4 Configuring the Pivot Item Labels When a report is created, a small list of default pivot items is created for the user. These items will almost certainly have to be modified to assign labels to the specific items, based upon the structure of the message. The configuration of the pivot item labels is performed as follows: 1. Click on the "Edit" button to the left of the "Pivot Test Report" entry, created in the above steps. This redisplays the "Pivot Report Edit" screen. 2. On the "Pivot Report Edit" screen, click the "Config" button next to the "Pivot Field Specifications" entry. This displays the "Pivot Item Configuration Screen" depicted in Section 2 of this manual. 3. The "Pivot Item Configuration Screen" displays two sample messages broken down by words. Visually inspect the sample messages, and identify the Source IP address of the HTTP request. (This will be the second IP address of the message, the first IP address being the address of the HTTP server, in this case ) Label this field "SrcIP". 4. Similarly, visually identify the HTTP Method (GET, POST, HEAD, etc.). Label this field "Method". 5. Similarly, visually identify the URL portion of the message. Label this field "URL". 6. Similarly, visually identify the Status code portion of the message. Label this field "Status". 7. Optionally eliminate the other fields of the message, so that the only labels displayed are the "SrcIP", "Method", "URL, and "Status" fields (with all other fields blank.) 8. Click "Continue" to return to the "Pivot Report Edit" screen, and then click "Save" to save the entire report settings. Note that the configuration of pivot items, described above, can also be performed when the report is created (as in step 3 above). In this example, the configuration of pivot item labels is performed as a separate step. Step #5 Generating And Viewing The Pivot Report Once a Pivot report is created, it is automatically generated each night after midnight, to process the data of the previous day. The user can also generate the reports manually by clicking the "Generate" button at the top of the screen (to Pivot Report Application Notes, Page - 30

31 generate all reports), or by clicking into a report and clicking the "Generate" button from that location (to generate an individual report.) When the user clicks the "Generate" button, the Pivot report is overwritten with new information, discarding the previous report and replacing it with a new report containing data up to the current time. After generating the Pivot Report, a series of hyperlinks will appear on the toplevel Pivot screen for each label created in step 4 above. In the case of this specific example, the "URL" report for the "Pivot Test Report" (accessed by clicking the "URL" hyperlink beneath the "Pivot Test Report" entry) will appear similar to the depiction below: The actual Pivot Report provides a list of all the values for each Pivot Item, and depicts a bar chart showing the distribution occurrence counts for each item. (For example, as shown above, the most common URL is the " value, and the distribution of all URLs in the Pivot Test Messages follows.) The operator can drill down into the hyperlink associated with each Pivot Item to see the actual messages that contained the pivot item. Additionally, the operator Pivot Report Application Notes, Page - 31