Custom Integrations

A custom integration lets you bring logs from a source that the built-in integrations do not cover, such as an in-house application or a device without its own card. You create the integration in UTMStack, open a listener for it on a UTMStack Collector, point your source at that listener, and add a pipeline that parses the new data.

How it works

  • When you create a custom integration, UTMStack derives an identifier from its name, for example acme-firewall for "Acme Firewall". This identifier is the data type of every event the integration receives.

  • The UTMStack Collector (the UTMStack Forwarder service on a Linux host in your network) listens for your source on the protocol and port you choose: syslog over UDP, TCP or TLS, or HTTP or HTTPS requests.

  • The Collector sends the events to UTMStack on port 443. The events are stored under your integration's data type.

  • UTMStack has no parser or rules for a new data type. You add a pipeline to turn the raw logs into fields, and then you can write alerting rules for that data type.

For how to install and run the Collector, see UTMStack Forwarder.

How the identifier is made

UTMStack turns the name into the identifier like this: letters become lower case, accents are removed (é becomes e), and every run of other characters (spaces, dots, slashes and so on) becomes one hyphen. Hyphens at the start and end are dropped.

Integration nameIdentifier and data typeCard title
Acme Firewallacme-firewallAcme Firewall
Billing API v2billing-api-v2Billing Api V2
JSON Inputjson-inputJson Input
SyslogsyslogSyslog

The card title is rebuilt from the identifier, so it may differ from what you typed in letter case.

Two names give data types that UTMStack already knows how to parse:

  • JSON Input gives json-input. The shipped pipeline json/json-input.yaml reads each event as one line of JSON, and 10 shipped rules use this data type. See JSON Input.

  • Syslog gives syslog. The shipped pipelines syslog/syslog-generic.yaml and syslog/syslog-cef-leef.yaml parse plain syslog, CEF and LEEF messages, and 3 shipped rules use this data type. See Generic Syslog, CEF and LEEF.

Any other name gives a new data type with no pipeline.

Before you start

  • You need the Administrator role (the integrations.write permission) to create integrations and to enable listeners on a Collector. Integrations are created in the tenant you are working in.

  • You need a Linux host for the Collector that your source can reach on the port you choose. One Collector can serve many integrations, so you can reuse an existing one.

  • Choose a port that no other integration on that Collector uses, and that no other program on the host uses.

Step 1: Create the custom integration

  1. 1

    Open the Integrations page

    Go to Configure > Integrations. On the Add custom integration card ("Create a new custom source"), click Create.

    New custom integration drawer in UTMStack with the Integration name, Description and Icon fields and the Create integration button
  2. 2

    Describe the integration

    In New custom integration, fill in:

    FieldWhat to enter
    Integration nameThe name of your source, for example Acme Firewall. Below the field, UTMStack shows the identifier it will store, for example "Stored as “acme-firewall” (derived from the name)".
    DescriptionOptional. What this integration ingests and why.
    IconOptional. A PNG, SVG, JPEG, GIF or WEBP image of up to 3 MB, used as the integration logo.

    If another integration already has the same identifier, UTMStack shows "That integration name can’t be used — an integration with that identifier already exists." and you cannot create it. Choose a different name.

  3. 3

    Create it

    Click Create integration. The new card appears on the Integrations page. Use the Custom button next to the category list to show only your custom integrations.

The identifier, and so the data type, is fixed when you create the integration. Later you can change only the description and the icon.

Step 2: Open a listener on a Collector

Open your custom integration's card. The setup guide shows a diagram of the flow (your source, the Collector and UTMStack) and the panel Enable this integration remotely.

  1. 1

    Pick a Collector

    In Collector, select the Collector that will receive the logs. The panel shows how many Collectors are online and offline. If you have no Collector yet, choose Add Collector: the guide opens Install the UTMStack Collector with the install command to run on a Linux host. See UTMStack Forwarder for details.

  2. 2

    Choose the protocol and port

    In Protocol, choose how your source sends data: UDP, TCP or TLS for syslog, or HTTP or HTTPS for requests. In Port, enter the port the Collector should listen on. The guide suggests port 7100.

    For a new data type, the Collector uses the port you enter the first time you enable it; to move it later, run change-port on the Collector host. If the identifier is a data type the Collector already knows (for example syslog), the Collector keeps that data type's own port and ignores the one you enter; see Generic Syslog, CEF and LEEF.

  3. 3

    Set the HTTP options (HTTP and HTTPS only)

    • Authentication: None, Bearer token or HMAC signature. With a token or signature, the Collector creates a secret and the panel shows it once ("Store this token securely — it won't be shown again."). Copy it then.

    • Path: the address path for incoming requests. If you leave it empty, the Collector uses /logs.

    • Signature header: for HMAC only, the header that carries the signature. If you leave it empty, the Collector uses X-Hub-Signature-256.

  4. 4

    Load a certificate (TLS and HTTPS only)

    TLS and HTTPS need a certificate on the Collector. Under TLS certificate, add the Certificate, the Private key and, optionally, the CA certificate, then click Upload certificate. If a valid certificate is already loaded, the panel says so.

  5. 5

    Enable the integration

    Click Enable integration. The panel then shows the current state, for example "Currently: enabled · udp · port 7100". If the Collector refuses the change, a message explains why (see Troubleshooting below).

An offline Collector cannot receive configuration changes. A Collector installed with remote control turned off ignores changes from the web interface; on that host, use the command line instead (below).

Enable the listener from the command line instead

Connect to the Collector host with SSH and run the command for your protocol and port. Replace acme-firewall with your integration's identifier.

sudo /opt/utmstack-forwarder/utmstack_forwarder enable-integration acme-firewall 7100 udp

The first time, the Collector prints New data type "acme-firewall" created. and then the port it listens on. The protocol can be udp, tcp, tls, http or https. For HTTP and HTTPS, you can add --auth bearer (token) or --auth hmac (signature). The Collector prints the new token once and keeps it in /opt/utmstack-forwarder/certs/integration-http-<identifier>.token.

For TLS and HTTPS, load your certificate and key first (once per Collector). An optional CA certificate can be added as a third argument; without it, the certificate itself is used as the CA.

sudo /opt/utmstack-forwarder/utmstack_forwarder load-tls-certs /path/to/cert.crt /path/to/key.key

To list the data types the Collector knows, including yours, run sudo /opt/utmstack-forwarder/utmstack_forwarder list-datatypes.

Step 3: Point your source at the Collector

Syslog (UDP, TCP or TLS): configure your source to send syslog to the Collector's address and the port you chose, for example <collector-ip>:7100. Most devices have a remote syslog or log forwarding setting for this.

  • Over UDP, the Collector reads at most 2,048 bytes of each message; longer messages are cut. Use TCP or TLS for sources that send long messages.

  • Over TCP and TLS, send one event per line. A line that starts with a digit is read as a length prefix (the framing of RFC 5425), so start each plain line with a syslog header such as <14>, or with any other character that is not a digit.

HTTP or HTTPS: send each event as the body of an HTTP POST request to the listener. The body can be JSON or any other text, up to 8 MB per request; one request becomes one event. For example, on the Collector host:

curl -k -X POST "http://127.0.0.1:7100/logs" 
  -H "Content-Type: application/json" 
  -d '{"message":"user jdoe logged in","host":"app-01","user":"jdoe"}'

If you chose Bearer token, add the header Authorization: Bearer <token>. If you chose HMAC signature, your source must compute an HMAC-SHA256 of each request body with the secret and send it as hexadecimal in the signature header, optionally with a prefix such as sha256=.

The listener answers each request with an HTTP status:

StatusMeaning
204Accepted.
401 unauthorizedThe bearer token or the HMAC signature is missing or wrong.
405 method not allowedThe request was not a POST.
413 payload too largeThe body is over 8 MB.
503 service unavailableThe Collector's queue is full. Wait the number of seconds in the Retry-After header (5) and send again.

An HTTP or HTTPS listener for a custom integration listens only on 127.0.0.1, the Collector host itself. Post from an application on that host, or put a reverse proxy in front of it. For sources on other machines, use syslog over UDP, TCP or TLS.

Step 4: Check that data arrives

  1. 1

    Send a test event

    Send one event from your source, or a test line from the Collector host. For UDP:

    logger --udp --server 127.0.0.1 --port 7100 "utmstack custom integration test"

    For TCP, use --tcp instead of --udp. For HTTP, use the curl command above.

  2. 2

    Open the logs

    Open the integration's card and click View logs. Log Explorer opens with your data type for the last 24 hours. Events usually appear within a minute. Until you add a pipeline, the useful text is in the raw field.

  3. 3

    Check the data source

    Within about 20 minutes, the sender appears on Configure > Data Sources under Direct. For syslog, the data source is named after the sender's IP address (or the Collector's host name for messages sent from the Collector itself). For HTTP and HTTPS, it is named after the data type.

Step 5: Parse the logs and add rules

Incoming events are stored under your integration's data type, but they are not parsed yet (unless you used one of the names in the tip at the top).

  1. Add a pipeline for the data type in Configure > Pipelines so the raw payload is mapped to the UTMStack standard event fields. The card's Filters button opens Pipelines filtered to your data type. See Pipelines and Pipeline File Format and Step Reference.

  2. Test the pipeline with a real sample before you save it. See Testing Pipelines and Rules.

  3. Write alerting rules for the data type in Configure > Alerting Rules. The card's Rules button opens the rules for your data type. See Alerting Rules and Rule File Format.

Worked example: an application that sends JSON lines over TCP

An in-house billing application writes one JSON object per line, for example:

{"ts":"2026-11-12T09:30:00Z","user":"jdoe","src":"203.0.113.10","event":"login","result":"failed"}
  1. Create a custom integration named Billing App. Its data type is billing-app.

  2. Enable it on your Collector with Protocol TCP and Port 7101.

  3. Configure the application, or a log shipper on its host, to send each line to <collector-ip>:7101 over TCP. The lines start with {, so they are read one per line.

  4. Create a pipeline for billing-app in Configure > Pipelines (Code mode):

pipeline:
  - dataTypes: [billing-app]
    steps:
      - json:
          source: raw
          where: 'startsWith("raw", "{")'
      - rename:
          from:
            - log.user
          to: origin.user
      - rename:
          from:
            - log.src
          to: origin.ip
      - rename:
          from:
            - log.event
          to: action

The json step writes each key as log.<key> (for example log.user), and the rename steps move the values to standard fields that rules, dashboards and threat intelligence read. See Standard Event Schema for the field names, and for how to write actionResult (it must be success, failed or denied).

  1. Test it with the sample line, save it, and check new events in Log Explorer: origin.user and origin.ip are filled in and the errors field is empty.

Troubleshooting

SymptomLikely causeFix
"That integration name can’t be used — an integration with that identifier already exists."Another integration, built-in or custom, has the same identifier.Choose another name. Check the identifier shown under the field.
Enabling fails with "port ... is already in use by integration '...'"Another integration on that Collector already listens on the same port and protocol.Choose another port, or disable the other integration.
Enabling fails with "port ... is already in use by another process"Another program on the Collector host uses the port, for example a syslog service on 514.Choose another port, or stop that program.
Enabling fails with "TLS certificates not found. Please load certificates first"TLS or HTTPS was chosen but no certificate is loaded on the Collector.Upload the certificate and key in TLS certificate, or run load-tls-certs.
"This Collector is offline and can't receive configuration changes right now."The Collector has no connection to UTMStack.Check the UTMStackForwarder service on the host. See UTMStack Forwarder.
No events, and the source is on another machineA firewall blocks the port, the source sends to the wrong address or protocol, or the listener is HTTP or HTTPS (local only).Open the port on the Collector host and in between. Match the protocol. Use syslog for remote sources.
HTTP requests get 401 unauthorizedThe token or signature is missing or wrong.Send Authorization: Bearer <token>, or the HMAC of the exact body in the signature header.
Long syslog messages arrive cutUDP messages are limited to 2,048 bytes.Switch the source and the listener to TCP or TLS.
A TCP stream stops after some linesA line started with a digit and was read as a length prefix.Make each line start with a syslog header such as <14>.
Events arrive but all text is in rawNo pipeline exists for the data type yet.Add a pipeline (Step 5).
Fields are missing after you added a pipelineA step failed. Failed steps write a message to the event's errors field and the next steps still run.Open an event in Log Explorer, read errors, and test the pipeline with that event. See Testing Pipelines and Rules.

Edit, disable or delete a custom integration

  • Edit: hover over the card and click the pencil icon. You can change the description and the icon.

  • Stop receiving logs: open the card, select the Collector in Enable this integration remotely and click Disable integration. Do this on every Collector where it is enabled.

  • Delete: hover over the card, click the bin icon and confirm with Delete. Deleting the card does not close listeners on your Collectors, so disable the integration first. To remove the data type from a Collector after disabling it, run sudo /opt/utmstack-forwarder/utmstack_forwarder remove-datatype <identifier> on the host. Events already stored stay searchable. Built-in integrations cannot be edited or deleted.