Getting StartedHow Data Reaches UTMStack

How Data Reaches UTMStack

UTMStack v12 receives logs in four ways: agents on your hosts, the UTMStack Forwarder for network devices and appliances, plugins that pull from cloud services, and two dedicated collectors. This page explains each one, follows a log from the source to storage, and shows how to use the Integrations page to set sources up.

The four ways in

Way inWhat it isUsed forBadge on the card
AgentA service you install on each Windows, Linux or macOS host. It reads the host's own logs and sends them to UTMStack.Windows, Linux and macOS hostsAgent
UTMStack ForwarderA Linux service that listens for logs sent by other systems (syslog, NetFlow, HTTPS webhooks) and forwards them to UTMStack. The web interface calls it the Collector.Firewalls, switches, hypervisors, antivirus consoles, Suricata, Oracle, GitHub, any syslog or CEF source, your own applicationsCollector
PluginUTMStack itself connects to a cloud service's API and pulls the logs. You only enter credentials.AWS, Azure, Google Cloud, Microsoft 365, CrowdStrike, Sophos CentralPlugin
Dedicated collectorA Linux service built for one source.IBM AS/400 (IBM i), and UTMStack's own container logsCollector

Agents, the Forwarder and the dedicated collectors all talk to the UTMStack server on port 443 only, over a direct connection (they have no proxy setting). See Ports and Network Requirements.

In UTMStack v12 the agent no longer receives syslog. Every device that sends syslog now goes to a UTMStack Forwarder, which you install once and reuse for many integrations. See UTMStack Forwarder.

Which way to choose

Your sourceUse
A Windows, Linux or macOS computer you can install software onThe agent for that system. It keeps logs on disk during outages and allows response commands.
Windows computers where you cannot install softwareWindows Event Forwarding to one Windows server that runs the agent.
A device or appliance that sends syslog, and has a cardThat card, through a Forwarder.
A device that sends syslog, CEF or LEEF, without a cardGeneric Syslog, CEF and LEEF, through a Forwarder.
An application that writes JSONJSON Input, through a Forwarder.
A cloud service with a cardThat card (plugin).
Anything elseCustom Integrations.

The path of a log

  1. 1

    Collection

    The agent reads the host's logs, the Forwarder receives logs from devices, or a plugin pulls them from a cloud API. Each log gets a data type (for example wineventlog or firewall-fortigate-traffic) and a data source (the host name, the sender's IP address, or the config group name for cloud integrations).

  2. 2

    Transport and identity

    Agents and collectors send their logs over port 443, on an encrypted connection, to the UTMStack server. Each one first registered with the server using the connection key in its install command, and received its own credentials. The server uses those credentials to decide which tenant the logs belong to. Plugins run on the server itself and use the tenant of their config group.

  3. 3

    Intake and queue

    The log intake service confirms each log only after it is safely in a queue. Agents keep a log on disk until it is confirmed and send it again if needed; the Forwarder does not keep logs on disk. The queue holds logs for up to 24 hours while the event processor works through them.

  4. 4

    Parsing

    The event processor runs the pipelines for the log's data type. They split the raw text into fields and map them to the standard field names (for example origin.ip, target.user, action). A step that fails writes a message to the event's errors field. See Pipelines.

  5. 5

    Analysis and detection

    The event is stored, checked against the alerting rules for its data type and against threat intelligence, and used to update the User Auditor. A match raises an alert. See How UTMStack Processes Events.

  6. 6

    Storage

    Events are stored in the log store, where Log Explorer, dashboards and compliance reports read them. They are kept for the retention period set under Data Retention and Cold Storage.

  7. 7

    Statistics

    Every 10 minutes, UTMStack counts what it received, parsed, analysed and stored, per data type and data source. These counts feed Configure > Data Processing and keep the Data Sources page up to date.

Is a source healthy?

QuestionWhere to look
Is the source still sending?Configure > Data Sources: Live (seen in the last 15 minutes), Idle (in the last 24 hours) or Offline. See Data Sources Page.
Are its logs arriving now?The card's View logs button, or Log Explorer with its data type.
Are its logs parsed?Expand an event in Log Explorer: standard fields are filled and errors is empty. Dropped (parsing) on Configure > Data Processing stays low.
Is the collector itself working?Agents: Updates and Status. Forwarder: the Collector list in any device guide shows online and offline Collectors. Cloud: the config group shows on the card, and its data source is Live.

For a step-by-step check when something is wrong, see Checking and Troubleshooting Data Collection.

The Integrations page

Go to Configure > Integrations. The page shows one card per integration.

  • Search integrations… filters the cards by name.

  • The category list (All categories) narrows the cards to one category, for example Firewall or Cloud.

  • Custom shows only the integrations your team created.

  • The three agents (Windows, Linux, macOS) always come first.

  • The last card, Add custom integration, lets you create your own source with Create. See Custom Integrations.

Each card shows its badge (Agent, Collector or Plugin) and a button:

ButtonMeaning
Set upOpen the setup guide.
EnabledA Plugin integration that has at least one saved config group in your current tenant.

Agent and Collector cards always show Set up, even while logs are arriving, because their setup lives on your hosts and not in UTMStack. Use the Data Sources page to see what is sending.

There is no "activate" switch in UTMStack v12. A device integration starts when you enable it on a Collector, an agent starts when you install it, and a cloud integration starts when you save a config group.

Inside an integration

Click a card to open its drawer. The top of the drawer shows the category, the badge and four buttons:

ButtonWhat it opens
Open dashboardThe dashboard list, searched for dashboards that belong to this integration.
View logsLog Explorer, filtered to this integration's data type for the last 24 hours.
RulesAlerting Rules, filtered to this integration's data type.
FiltersPipelines (parsing filters), filtered to this integration's data type.

Below the buttons is the setup guide:

  • Agents show the install requirements, the install command for each platform, and an Uninstall the agent section.

  • Collector integrations that use the Forwarder start with Enable this integration remotely, then the steps for the device, then Install the UTMStack Collector, Optional — Enable via the command line instead and Optional — Uninstall the Collector.

  • Plugin integrations show a credentials form. Each connection is saved as a config group with a name you choose (the v11 button "Add tenant" is now Add config group). The config group name becomes the data source name on the Data Sources page and cannot be changed later. UTMStack checks the credentials when you save (except for Sophos Central).

  • As 400 (IBM AS/400) and Utmstack (self-monitoring) show the install guide of their own collector.

Start of a Collector setup guide (FortiGate): How it works, then the Enable this integration remotely panel

Data types

Every integration writes its logs under a data type, for example firewall-cisco-asa or wineventlog. The data type decides which pipeline parses the logs and which detection rules apply. You see it in Log Explorer, on the Data Sources page, and in the commands that enable an integration on the Forwarder.

A device sends to the Forwarder port of its integration, and that port decides the data type. A device that sends to the wrong port is stored under the wrong data type and is not parsed correctly.

Built-in integrations

UTMStack v12.0.2 ships 34 built-in integrations. Every tenant sees them; tenants cannot edit or delete them. The card names below are the ones UTMStack v12.0.2 shows.

CardBadgeCategoryData typeSetup page
Windows AgentAgentOperating SystemwineventlogWindows Agent
Linux AgentAgentOperating SystemlinuxLinux Agent
MacosAgentOperating SystemmacosmacOS Agent
VmwareCollectorVirtualizationvmware-esxiVMware ESXi
NetflowCollectorNetworknetflowNetFlow
Cisco SwitchCollectorNetworkcisco-switchCisco Switch
CiscoCollectorFirewallfirewall-cisco-asaCisco ASA
Fire PowerCollectorFirewallfirewall-cisco-firepowerCisco Firepower
MerakiCollectorFirewallfirewall-merakiCisco Meraki
FortigateCollectorFirewallfirewall-fortigate-trafficFortinet FortiGate
FortiwebCollectorFirewallfirewall-fortiwebFortinet FortiWeb
Palo AltoCollectorFirewallfirewall-paloaltoPalo Alto Networks
Sonic WallCollectorFirewallfirewall-sonicwallSonicWall
Sophos XgCollectorFirewallfirewall-sophos-xgSophos Firewall (XG)
PfsenseCollectorFirewallfirewall-pfsensepfSense
MikrotikCollectorFirewallfirewall-mikrotikMikroTik
SuricataCollectorIDS / IPSsuricataSuricata
AixCollectorOperating Systemibm-aixIBM AIX
As 400CollectorOperating Systemibm-as400IBM AS/400 (IBM i) Collector
OracleCollectorDatabaseoracleOracle Database
UtmstackCollectorSIEMutmstackUTMStack Self-Monitoring
EsetCollectorAntivirusantivirus-esmc-esetESET PROTECT
KasperskyCollectorAntivirusantivirus-kasperskyKaspersky Security Center
Sentinel OneCollectorXDRantivirus-sentinel-oneSentinelOne
Deceptive BytesCollectorDeceptiondeceptive-bytesDeceptive Bytes
GithubCollectorDevOpsgithubGitHub
AWS CloudwatchPluginCloudawsAWS CloudWatch
AzurePluginCloudazureMicrosoft Azure
GcpPluginCloudgoogleGoogle Cloud
O365PluginCloudo365Microsoft 365 (Office 365 Management Activity API)
O365 GraphPluginCloudo365-graphMicrosoft 365 through Microsoft Graph
CrowdstrikePluginXDRcrowdstrikeCrowdStrike Falcon
SophosPluginXDRsophos-centralSophos Central
BitdefenderPluginAntivirusantivirus-bitdefender-gzNot yet documented for v12

Sources without a card

SourceData typePage
Any syslog, CEF or LEEF sendersyslogGeneric Syslog, CEF and LEEF
Applications that write one JSON object per linejson-inputJSON Input
Windows computers forwarding to a collectorwineventlogWindows Event Forwarding
Your own sourcesA data type you nameCustom Integrations

Permissions

ActionPermission needed
See the Integrations page and the setup guidesintegrations.read
Save cloud credentials, enable or disable an integration on a Collector, see the list of Collectorsintegrations.write
See the connection key inside the install commandsAdministrator role

Install commands contain your instance's connection key. If you are not an administrator, the key appears as ******* and the command will not work until an administrator gives you the real key. The key is managed under Configure > Settings > Connection Key.

Always copy install commands from your own UTMStack instance. They contain your server address and connection key. Do not share them.

Multi-tenant installations

With the multi-tenant licence, each tenant has its own integrations, config groups, agents and Collectors. Agents and Forwarders belong to the tenant whose connection key was in their install command, and every log they send is stored in that tenant. When an operator has entered a child tenant, the install commands use that tenant's domain as the server address instead of the address in the browser. Copy commands from inside the tenant the host belongs to. See Multi-Tenancy Overview.