Filter Steps Reference

This page provides a detailed reference for all 12 transformation steps available in the EventProcessor parsing pipeline.

1. json

Parses a JSON string and extracts its keys.

• Fields: source (Required), where (Optional CEL condition).

• Placement: All extracted keys are automatically prefixed with log. (e.g., {"id": 1} becomes log.id: 1).

Example:

- json: 
    source: raw
    where: 'contains(raw, "{")'

2. rename

Maps existing fields to new names.

• Fields: from (Array of source paths), to (Target path), where (Optional).

• Behavior: Moves the value from the source path to the target path.

Example:

- rename:
    from: [log.user_name, log.login]
    to: origin.user
    where: 'exists("log.user_name")'

3. cast

Converts field types.

• Fields: fields (Array), to (Target type), where (Optional).

• Supported Types: int, float, string, bool, []string.

Example:

- cast:
    fields: [origin.port]
    to: int

4. delete

Removes fields from the log to optimize storage and indexing.

• Fields: fields (Array), where (Optional).

Example:

- delete: 
    fields: [log.temporary_header, log.internal_id]

5. grok

Pattern matching for unstructured text.

• Fields: source (Defaults to raw), patterns (List of { fieldName, pattern }), where (Optional).

• Placement: Uses the fieldName provided in the pattern list literally. To use standard namespaces, specify them (e.g., origin.ip). By convention, custom fields should use log..

• Extensibility: Users can add or modify standard patterns through the UTMStack WebUI.

Default Standard Patterns

Example:

- grok:
    source: raw
    patterns:
      - fieldName: origin.ip
        pattern: '{{.ipv4}}'
      - fieldName: log.event_id
        pattern: 'ID: {{.integer}}'

6. kv (Key-Value)

Extracts key-value pairs from a string.

• Fields: source, fieldSplit (Separator between pairs), valueSplit (Separator between key and value), where (Optional).

• Placement: Like the json step, all extracted keys are automatically prefixed with log..

Example:

- kv: 
    source: raw
    fieldSplit: " " 
    valueSplit: "="

7. trim

Cleans strings by removing prefixes, suffixes, or matching patterns.

• Fields: fields (Array), function (prefix, suffix, substring, regex), substring (the string or pattern to trim), where (Optional).

Example:

- trim: 
    function: suffix
    substring: ".local"
    fields: [origin.host]

8. add

Injects a fixed value into a field.

• Fields: function (string), params (Map), where (Optional).

• Required Params: key (target path), value (the actual value to add).

• Placement: Uses the key literally.

Example:

- add: 
    function: string
    params: 
      key: log.category
      value: security

9. reformat

Converts field formats, primarily for timestamps.

• Fields: fields (Array), function (time), fromFormat (Go layout), toFormat (Go layout), where (Optional).

• Behavior: Overwrites the value in the specified fields.

Example:

- reformat:
    fields: [deviceTime]
    function: time
    fromFormat: 'Jan 02 15:04:05'
    toFormat: '2006-01-02T15:04:05Z'

10. csv

Parses comma-separated values.

• Fields: source, separator, headers (Array of target paths), where (Optional).

• Placement: Uses the names provided in headers literally.

Example:

- csv:
    source: raw
    separator: ","
    headers: [log.id, origin.user, action, actionResult]

11. dynamic

Calls an external gRPC plugin.

• Fields: plugin (Name), params (Key-value map), where (Optional).

Example:

- dynamic:
    plugin: com.utmstack.geolocation
    params: 
      source: origin.ip
      destination: origin.geolocation

12. drop

Discards the log immediately.

• Required: where (CEL condition).

Example:

- drop:
    where: equals("origin.ip", "127.0.0.1")