Pipelines

A pipeline tells UTMStack how to read one kind of raw log and turn it into a standard event. On Configure > Pipelines you see every pipeline, switch pipelines on or off for your tenant, change the order in which they run, and write your own. UTMStack v11 called pipelines "filters"; the shipped files are still kept in a folder named filters.

Before you start

• The Configure section of the menu is shown to administrators only.

• Viewing the page needs the permission eventprocessing.read.

• Creating, editing, deleting, switching on or off, reordering and testing need eventprocessing.write. Of the built-in roles, only Administrator has it. See Roles and Permissions.

The pipeline list

The page is titled Pipelines: "YAML pipelines applied to logs as they're parsed during ingestion."

Columns:

Switch a pipeline on or off

Use the Active switch in the pipeline's row. The change applies to your tenant only; other tenants keep their own setting. It works for shipped and custom pipelines, and reaches the engine within about a minute.

Change the order

When several pipelines process the same data type, they run one after the other, and each step sees what the previous pipelines wrote. Use Move up and Move down to change the sequence.

• Your order is saved for your tenant only. The shipped files are not changed, and other tenants keep their own order.

• The engine runs the pipelines in the order you set and then any remaining ones by the order value in their files.

• Order matters only between pipelines that share a data type. In the shipped set, only syslog has more than one pipeline: the generic syslog pipeline runs first, then the CEF and LEEF stages.

View a shipped pipeline

Click a system pipeline to open it. The panel shows: "System pipeline — read-only. You can only enable or disable it." You can read its YAML and test it, but not change or delete it.

To change what a shipped pipeline does, either add a custom pipeline for the same data type that runs after it and adjusts its result, or switch the shipped pipeline off and create a custom pipeline that replaces it. There is no import, export or copy function for pipelines.

Create a custom pipeline

Write a custom pipeline to parse a source that UTMStack does not cover, for example the data type of a custom integration (see Custom Integrations), or to add fields to a source it does cover.

1. 1

   Open the editor

   Click New pipeline. Enter a Name. The editor shows the file name it will use ("Will be saved as …"): your name in lower case, with spaces and other characters replaced by _.

2. 2

   Describe the pipeline

   Use the Visual tab or the Code tab. Both edit the same pipeline.

   In Visual, each pipeline entry has:

   • Data types: the data types it processes. Pick existing ones or type a new name.

   • Order: filled in as one more than the highest order already in use.

   • Steps: choose a step type under Add step: (grok, kv, json, csv, trim, rename, cast, reformat, delete, drop, add or dynamic). grok, json, kv, rename, add and drop have their own forms; the others are edited as YAML ("This processor is edited as raw YAML."). Each step has an optional condition (CEL).

   Add pipeline entry adds another entry to the same file, for example for a second data type.

   In Code, write the YAML directly. Insert regex pattern inserts one of the built-in named patterns at the cursor. See Pipeline File Format and Step Reference for the format, and Pipeline Examples for complete pipelines to paste.

   

   The picture is from UTMStack v12.0.0. In UTMStack v12.0.2 each pipeline entry is headed "Entry 1", "Entry 2" and so on, and also has the Order box and the Add pipeline entry button described above.

3. 3

   Test it

   Click Test, choose a data type, paste a sample raw log and click Run. The test runs only the pipeline in the editor. See Testing Pipelines and Rules.

4. 4

   Save

   Click Save. The page confirms "Pipeline saved". The pipeline is active for your tenant and reaches the engine within about a minute.

Your pipeline is saved as pipeline/filters/user/<tenant id>/<name>-<tenant id>.yaml in the engine work folder. UTMStack writes your tenant id into each entry, so the pipeline only runs on your tenant's logs.

What saving checks

When you save, UTMStack checks that:

• the pipeline has at least one entry;

• each entry has data types and at least one step;

• each step has exactly one known step type and its required fields (for example source for json, from and to for rename);

• each grok pattern has both a field name and a pattern.

An entry with no order, or with 0, gets order 100.

When it saves, UTMStack writes the file again with your tenant id in each entry. The result runs the same, but comments are removed and keys may change order. Keep your own commented copy of a pipeline if you need the comments.

Edit or delete a custom pipeline

Open the pipeline from the list. Custom pipelines can be edited, tested and deleted. Delete asks you to confirm.

Apply a pipeline to all tenants

On a multi-tenant installation, the instance operator also sees Apply to all tenants in the pipeline panel and the pipeline list. In UTMStack v12.0.2 it works for:

• creating a new custom pipeline in the selected tenants (each tenant gets its own copy), and

• switching a shipped pipeline on or off in the selected tenants.

Changing, deleting or switching a custom pipeline reaches only your own tenant, because each tenant's copy is a separate file. To change the copies in other tenants, enter each tenant and edit its copy there. Failures on single tenants are listed after the run and are not rolled back. See Apply to All Tenants.