How to Find What You Want Using simple regex in HPE ArcSight Logger

Size: px

Start display at page:

Download "How to Find What You Want Using simple regex in HPE ArcSight Logger"

Daisy Rice
5 years ago
Views:

The piece that you want is either within a field already, or is not in a field in the Logger schema.

1 Introduction HPE ArcSight SmartConnectors and FlexConnectors do a great job parsing, normalizing, and categorizing events. Sometimes there is some additional parsing that you might want to do, to get your hands on part of a value to chart it, or sum it up, or count it up. The piece that you want is either within a field already, or is not in a field in the Logger schema. When you find yourself in this situation, Regular Expressions (or RegEx) can be used to isolate the part that you want from the parts that you do not want. The technique described here can be used for just about any piece of an event. The Use Case we will use here is to see what versions of Connectors are sending events in to ArcSight, so we know what versions are out there, and we can determine if, and where, we might need to upgrade Connectors. In the Common Event Format schema, the agentversion field has a short name of av. It is well known that a chart of Top Names or Top Source Address can be produced using the ArcSight Logger Search/Analyze feature. Here are those searches: top name top sourceaddress In the 2 examples above, both the name and the sourceaddress field are part of the ArcSight Logger schema and are both indexed fields by default. This can be seen in Logger s Default Fields display and seeing both name and sourceaddress listed, and indexed. With Logger 6.2, type default into the Take me to navigation. By default the agentversion field is not included in the indexed fields. If you try to run a search and summarize using the agent version field av, the result is the following: The field agentversion could be added to the indexes, but before any field is added to the indexes, consider how often might you search on agentversion? The decision to add this field to the indexing adds some small amount of indexing work to Logger. Since we do not expect to always be searching on agentversion, we will use some RegEx to do our work instead and not add this work to the Logger indexer. As a result, the search may not be as fast as an indexed search, but the tradeoff is that nearly anything can be specified as part of the search.

2 Back to the Use Case: What are the varying agentversion values? If we can isolate the version numbers, we can use the top command to do the work for us. The next stage is to isolate the version numbers from the agentversion fields, and we use Logger Regular Expressions to do the work for us. Zeroing in on What We Want The Logger rex command specifies a pattern to look for values, against the event in Logger. Let s look at a real example. Here is a CEF event, and the agentversion field and value are highlighted. CEF:0 ArcSight ArcSight agent:050 Connector Raw Event Statistics Low eventid= mrt= catdt=security Mangement art= cat=/agent/rawevent/statistics deviceseverity=warning rt= filetype=agent cs1= cs2= cs3=0.14 cs4=6447 cs5=21.49 cs6=3uegky1ababdqg3pb+fslhg== cn1= cn2= cn3=42 devicecustomdate1= c6a4=fe80:0:0:0:20c:29ff:fe41:91b4 cs1label=event throughput cs2label=raw event character throughput cs3label=event throughput (SLC) cs4label=raw event length (SLC) cs5label=raw event character throughput (SLC) cs6label=destination ID cn1label=total event count cn2label=total raw event length cn3label=event count (SLC) devicecustomdate1label=last time c6a4label=agent IPv6 Address ahost=romeo.foobar.com agt= agentzoneuri=/all Zones/ArcSight System/Private Address Space Zones/RFC1918: av= atz=us/eastern aid=3uegky1ababdqg3pb+fslhg== at=syslog dvchost=romeo.kramerapps.com dvc= devicezoneuri=/all Zones/ArcSight System/Private Address Space Zones/Local Network Zones/RFC5735: IANA - Loopback ( ) dtz=us/eastern _cefver=0.1 The rex command lets us specify a pattern to look for the agentversion in the CEF event. In the event, the agentversion field and value is specified by av= The name of the field is av and the value is The rex command also allows us to isolate and capture whatever follows that equals sign =, i.e. the numbers and decimals. The capture value is whatever is contained within parenthesis specified in the search. Let s start crafting our search. Any event coming in from Connectors will have an agentversion value. Connector events are from ArcSight Products, so we start with this as our initial search. deviceproduct=arcsight Next, we add our regular expression pattern search, using the rex command. Remember, we are looking for that special string of av=some.numbers.and.digits.we.want.to.capture deviceproduct=arcsight rex "av=(?<av>[^\s]+)\s" Let s dissect the search command above, piece by piece.

3 Following the initial search is a character, also called the pipe character. This means take all the events from the left hand side of the, now known as the output of the first part of the search, and direct them at the command on the right hand side of the command, now known as the input to the next part of the search. Next is the rex command itself. This is the word rex followed by a set of quotes. deviceproduct=arcsight rex "" Next is the pattern that we are looking for. We want to pull out whatever is after the av= equals sign. In this next step of building our command, we tell rex to look for the pattern av= and to extract what follows the equals sign. deviceproduct=arcsight rex "av=()" Let s keep building. Since we want to refer to the agentversion value later on, we can give these new values a name. The name we choose goes inside the angle brackets like this <av> 1. deviceproduct=arcsight rex "av=(?<av>)\s" This new variable name could be nearly anything you wish and does not need to match the av we are searching on. This would also work: deviceproduct=arcsight rex "av=(?<myav>)\s" A? is included to capture the match into a backreference now called av. We re almost done. deviceproduct=arcsight rex "av=(?<av>[^\s]+)\s" We complete the rex command by adding a few symbols that mean anything except a space, one or more times 2. This is all followed by one last space, shown as \s. Since CEF key=values are separated by a space, this last space in our regular expression means the end of the agentversion key=value, separating it from the next key=value in the CEF event. Putting it all together in the Logger search will extract out the values, assign them to a search-time field called av and display that column of values in Logger like this: 1 The syntax here is sometimes called a named backreference. 2 Specifying anything except a something is also known as a negating character class in RegEx parlance.

4 And now we can ask Logger to tally up this new field we created with the rex command, and show us the distribution of values.

5 Summary Using a live example the use of HPE ArcSight Logger rex command was used to build a regular expression search to pick out a part of an event and summarize events using the isolated pieces of events. The pieces do not have to be added to Logger schema, and do not have to be added to the Logger indexing in order to summarize the event activity. For further information: - ArcSight Logger Administrator s Guide, Logger 6.2, page

Introduction to Scratch

Introduction to Scratch Familiarising yourself with Scratch The Stage Sprites Scripts Area Sequence of Instructions Instructions and Controls If a computer is a box think of a program as a man inside the