How to create filters in data parsing?

UTMStack allows you to customize filters in data parsing:

  1. Click on the hamburger positioned in the upper right part of the dashboard. Select the option Data
  2. In this panel, you will see all the filters implemented by UTMStack. Click on the button Add filter to create your own.filters-ok.jpg

UTMStack uses the Grok filter plugin from Logstash to match a line against a regular expression, map specific parts of the line into dedicated fields, and perform actions based on this mapping.

Basically, the syntax of Grok pattern is %{SYNTAX:SEMANTIC}.

The SYNTAX represents the pattern in the text of each log. The SEMANTIC is the identifier that you give to the piece of text in your parsed logs. Put simply, the Grok pattern represents %{PATTERN:FieldName}.

For example, a pattern like will match the Grok IP pattern.


UTMStack needs that the filters include the following output fields so that the correlation engine can analyze data.

  • @timestamp

The "@timestamp" is a JSON representation of the date and time where the log was entered to UTMStack. Its structure follows the pattern YYYY-MM-DDTHH:MM:SS.MsMsMsZ, which means "2017-01-18T11:41:28.753Z". You need to use Grok to extract that date string into a new field called timestamp.

  • dataType

Each log has a field dataType. This type indicates if the log is an AWS log, Mac OS log, o365 log, winEvent log, iis log, metric, NIDS, VMware ESXi, and NetFlow.

  • dataSource

This field represents the source where the log was generated.

Let's see a real filter example in UTMStack:

Input vmware-esxi log: 

This filter takes the vmware-esxi logs and validates that data input is by Syslog and does not contain data fields dataType. If the condition is true applies a Grok plugin filter, matching the message with the log structure. 

Then it's applied a condition to validate if the log is the vmware-esxi type, which, if true, removes the message field and adds the field dataType with the value "vmware-esxi".grok-filter-vmware-ok.jpg

After applying the filter, these are the output fields: