Microsoft Azure

The Azure integration brings activity from your Azure subscription into UTMStack. You stream subscription events into an Azure Event Hub; UTMStack reads the hub and saves its read position in an Azure Storage Account. Nothing is installed in your Azure tenant.

How it works

An Event Hub is an Azure service that receives a stream of events and keeps them for a retention time you choose, so that readers can collect them.

  • Your Azure subscription sends events through an Event Subscription (Event Grid) to an Event Hub. You can also send Azure Monitor and Microsoft Entra ID logs to the same hub with diagnostic settings (Step 3, optional part).

  • UTMStack reads the Event Hub with a connection string that has Listen rights, using its own consumer group.

  • UTMStack stores its checkpoints (how far it has read) in a container of a Storage Account. The first time it connects, it starts with the oldest event still kept in the hub. After that it continues from its checkpoint, also after a restart.

  • Events are stored with the data type azure and parsed by the shipped pipeline azure/azure-eventhub.yaml (version 2.3.0). UTMStack v12 ships 40 detection rules for this data type; see Azure in the Rules tab.

  • UTMStack supports Event Hubs in the Azure public cloud, Azure US Government and Azure China.

Before you start

  • In Azure, you need permission to create an Event Hubs namespace, a Storage Account and Event Subscriptions, and to register resource providers on the subscription.

  • In UTMStack, you need the Administrator role (the integrations.write permission). Configuration is saved in the tenant you are working in.

  • The UTMStack server must be able to reach Azure over HTTPS (outbound port 443).

Step 1: Create the Event Hub and its access policy

  1. 1

    Create an Event Hub

    In the Azure portal, create an Event Hubs namespace and an Event Hub inside it. UTMStack reads events from this hub.

    Follow the official Azure guide: Quickstart: Create an event hub using Azure portal.

  2. 2

    Add a shared access policy

    Open the Event Hub, go to Shared access policies and create a new policy. Make sure you select the Listen permission.

    Azure
  3. 3

    Copy the primary connection string

    Open the policy you just created and copy Connection string–primary key. You will paste it into UTMStack in Step 4.

    Azure

    The value looks like this (example values):

    Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=<policy name>;SharedAccessKey=<key>;EntityPath=<event hub name>

    UTMStack needs the EntityPath part, which names the Event Hub. If the string you copied has no EntityPath, add ;EntityPath=<event hub name> at the end.

  4. 4

    Create a dedicated consumer group

    Go to All services > Event Hubs > your namespace > Event Hubs > your Event Hub > Consumer groups, create a consumer group for UTMStack and note its name.

    Azure

    Create a consumer group only for UTMStack. Do not use $Default or any other group that is already in use: sharing a consumer group with unrelated readers can cause unexpected behavior and lost events. All UTMStack instances that read this hub should use the same consumer group so they can process events together.

Step 2: Create the Storage Account

  1. 1

    Create a Storage Account

    UTMStack needs a Storage Account to save its read position. Create one in the same region as the Event Hub.

    Follow the official Azure guide: Quickstart: Create a storage account.

  2. 2

    Get the storage container name

    Go to All services > Storage accounts > your account > Containers and note the name of the container UTMStack should use.

    Azure
  3. 3

    Copy the Storage Account connection string

    Go to All services > Storage accounts > your account > Access keys and copy the connection string with key.

    Azure

    The value looks like this (example values):

    DefaultEndpointsProtocol=https;AccountName=<storage account>;AccountKey=<key>;EndpointSuffix=core.windows.net

Step 3: Send subscription events to the Event Hub

  1. 1

    Register the Event Grid resource provider

    Go to Subscriptions > your subscription > Resource providers, search for Event Grid (Microsoft.EventGrid) and make sure it is registered.

    Azure
  2. 2

    Create an Event Subscription

    In the subscription's Events panel, click Event Subscription to create a new one.

    Azure
  3. 3

    Point the Event Subscription at your Event Hub

    Configure the Event Subscription to send events to the Event Hub you created in Step 1.

    Azure

Optional: send Azure Monitor and Microsoft Entra ID logs to the same hub

The Event Subscription above sends resource changes (writes, deletes and actions). Many Azure rules in UTMStack also read Microsoft Entra ID sign-in and audit logs, the Azure Activity log, Key Vault audit events, Application Gateway firewall logs and AKS (Kubernetes) audit logs. Azure sends these through diagnostic settings, which can stream to an Event Hub.

LogWhere to add the diagnostic setting in the Azure portalCategories the UTMStack pipeline recognises
Microsoft Entra IDMicrosoft Entra ID > Monitoring > Diagnostic settingsAuditLogs, SignInLogs, NonInteractiveUserSignInLogs, ServicePrincipalSignInLogs, ManagedIdentitySignInLogs, UserRiskEvents
Azure Activity logMonitor > Activity log > Export Activity LogsAdministrative
Key VaultThe vault's Diagnostic settingsAuditEvent
Application GatewayThe gateway's Diagnostic settingsApplicationGatewayFirewallLog, ApplicationGatewayAccessLog
AKS clusterThe cluster's Diagnostic settingskube-audit, kube-audit-admin

For each one:

  1. Click Add diagnostic setting and give it a name.

  2. Select the categories you want.

  3. Select Stream to an event hub, choose your subscription, the Event Hubs namespace and the Event Hub you created in Step 1. Choose the hub by name, so all logs go to the hub UTMStack reads.

  4. Click Save.

A diagnostic setting writes to the hub with a namespace-level policy (by default RootManageSharedAccessKey), which Azure requires to have Manage, Send and Listen rights. This is separate from the Listen-only policy you created for UTMStack. Microsoft Entra ID sign-in logs need a Microsoft Entra ID P1 or P2 licence.

Step 4: Connect Azure in UTMStack

Each Event Hub connection is a config group. You can add more than one.

  1. 1

    Open the integration

    Go to Configure > Integrations, search for Azure and open the Azure card. The form is at the end of the setup guide, under Event Hub configuration.

    Azure setup guide in UTMStack, showing How it works and the first Azure steps: create an Event Hub and add a shared access policy
  2. 2

    Fill in the form

    FieldWhat to enter
    Config group nameA name for this connection, for example acme-prod. It cannot be changed later, and it becomes the data source name on the Data Sources page.
    Event Hub connection stringThe connection string from Step 1. It must include EntityPath.
    Consumer groupThe consumer group you created for UTMStack in Step 1.
    Storage container nameThe container name from Step 2.
    Storage account connection stringThe Storage Account connection string from Step 2.

    All fields are required.

  3. 3

    Save

    Click Add config group. Before saving, UTMStack checks that it can reach the Event Hub, that the consumer group exists and that the storage container exists. When the check passes, you see "Config group created successfully."

If the save fails

When the check fails, the config group is not saved and a message starting with "credential verification failed:" explains why.

MessageWhat to check
Invalid Event Hub Connection String format. It must include an EntityPathAdd ;EntityPath=<event hub name> to the connection string.
Invalid Event Hub Connection String format. Please verify the Endpoint URL and SharedAccessKey are correct.Part of the string is missing or was cut when you copied it. Copy it again from the policy.
Event Hub Connection String is unauthorizedThe shared access policy is missing the Listen permission, or the key is wrong.
Event Hub ... was not foundThe EntityPath does not match the Event Hub name.
Consumer Group ... was not found on Event Hub ...The consumer group name is wrong, or it was created on another Event Hub.
Storage Container ... was not foundThe container does not exist in that Storage Account.
Cannot connect to Azure Storage / Access denied to Storage ContainerThe Storage Account connection string is wrong or has no access.
Connection to Event Hub timed out / Connection to Azure Storage timed outThe UTMStack server cannot reach Azure on port 443.

Check that events arrive

  1. 1

    Check that the hub receives events

    In the Azure portal, open your Event Hub. The Overview page shows charts of incoming messages. If there are none, Azure is not sending to this hub yet: check the Event Subscription and the diagnostic settings.

  2. 2

    Open the logs in UTMStack

    Open the Azure card again and click View logs. Log Explorer opens with the azure data type for the last 24 hours. UTMStack reads the hub continuously, so events usually appear within a few minutes of reaching the hub.

  3. 3

    Check the data source

    Go to Configure > Data Sources and open the Integrations tab. Your config group is listed by its name. The row is created when you save, and turns Live after the first events are counted, which can take up to 20 minutes.

Troubleshooting

Errors that happen after the config group is saved, for example a rotated key, do not appear in the web interface. They are written to the logs of the UTMStack event processor on the server, where the integration runs.

SymptomLikely causeFix
The hub shows incoming messages, but no azure events arrive in UTMStackThe connection string or storage key was rotated after you saved, or the consumer group was deleted.Edit the config group and paste the current connection strings, or create the consumer group again.
Some events are missingAnother program reads the hub with the same consumer group, so the two readers share the work.Give UTMStack its own consumer group, used by nothing else.
Resource changes arrive, but no sign-ins or audit eventsOnly the Event Subscription sends to the hub.Add the diagnostic settings for Microsoft Entra ID and the Activity log (Step 3, optional part).
Diagnostic setting logs go to a hub called insights-logs-...The diagnostic setting has no Event Hub name, so Azure created one hub per category.Edit the diagnostic setting and select your Event Hub by name.
Older events are missing after a long outageThe hub keeps events only for its retention time.Raise the retention time of the Event Hub if UTMStack may be unreachable for longer.
The Integrations row shows Offline although events arriveThe data source's "last seen" time is updated from the ingestion counts, which can take up to about 20 minutes.Wait 20 minutes and click Refresh.

Manage config groups

  • Add more connections: save one config group per Event Hub.

  • Edit: click the pencil icon next to a config group. The name cannot be changed. Secret fields (both connection strings) show a masked value; leave it as it is to keep the stored value. Click Update config group.

  • Delete: click the bin icon. The config group is deleted at once, without a confirmation, and collection stops. The data source it created stays on the Data Sources page; delete it there if you no longer need it.