Google Cloud
The Google Cloud integration brings your Google Cloud project logs into UTMStack. A Logs Router sink streams the logs into a Pub/Sub topic, and UTMStack pulls them from a subscription on that topic with a service account key. Nothing is installed in your Google Cloud project.
How it works
Pub/Sub is Google Cloud's messaging service: a topic receives messages, and a subscription keeps them until a reader collects and acknowledges them.
Cloud Logging sends your project logs through a Logs Router sink to a Pub/Sub topic.
UTMStack reads a Pull subscription on that topic, using the JSON key of a service account, and acknowledges each message once it has queued it. Messages that reached the subscription while UTMStack was not reading are collected when it reconnects, as long as Pub/Sub still keeps them.
Events are stored with the data type
googleand parsed by the shipped pipelinegoogle/gcp.yaml(version 2.3.1). UTMStack v12 ships 43 detection rules for this data type; see Google Cloud in the Rules tab.
Before you start
In Google Cloud, you need permission to create Pub/Sub topics and subscriptions, Logs Router sinks and service accounts in the project.
In UTMStack, you need the Administrator role (the
integrations.writepermission). Configuration is saved in the tenant you are working in.The UTMStack server must be able to reach Google Cloud over HTTPS (outbound port 443).
Step 1: Create the Pub/Sub topic and subscription
- 1
Create a Pub/Sub topic
Go to the Pub/Sub topics page in the Google Cloud console and click Create a topic. In Topic ID, enter a unique topic name and save. The Logs Router sink will publish your project logs to this topic.
- 2
- 3
Step 2: Route your logs to the topic
- 1
- 2
- 3
- 4
Choose which logs to send (optional)
Use inclusion and exclusion filters to choose which log entries are sent. If you do not set a filter, all logs of the project are sent to the topic. Click Create sink.
For example, this inclusion filter sends only Cloud Audit Logs, which most Google Cloud rules read:
logName:"cloudaudit.googleapis.com"
When you create the sink in the console and the topic is in the same project, Google gives the sink's writer identity permission to publish to the topic. If the topic is in another project, grant the sink's writer identity (shown on the sink's details) the Pub/Sub Publisher role on the topic.
Admin Activity audit logs are always written. Data Access audit logs are off by default for most Google Cloud services; turn them on in IAM & Admin > Audit Logs if you want them in UTMStack.
Step 3: Create a service account and key
- 1
Create a service account
Go to the service accounts page, select the project that holds your topic and sink, and click Create service account. UTMStack uses this account to pull logs from Pub/Sub.
- 2
Grant the Pub/Sub role
Fill in the form and give the service account the Pub/Sub Editor role so UTMStack can pull from the subscription.
When you save, UTMStack checks that the subscription exists, which needs permission to read the subscription's details. The Pub/Sub Subscriber role alone does not include that permission, so the check fails with it. If you do not want to grant Pub/Sub Editor on the whole project, grant Pub/Sub Subscriber and Pub/Sub Viewer on the subscription instead.
- 3
- 4
Step 4: Connect Google Cloud in UTMStack
Each project connection is a config group. You can monitor more than one project by saving more config groups.
- 1
- 2
Fill in the form
Field What to enter Config group name A name for this connection, for example acme-prod. It cannot be changed later, and it becomes the data source name on the Data Sources page.Project ID The ID of the Google Cloud project that holds your topic, subscription and sink. Subscription name The Pull subscription you created in Step 1. Service account JSON key Click Select file and choose the JSON key file from Step 3. All fields are required. UTMStack does not need the topic name.
- 3
Save
Click Add config group. UTMStack checks the key, the project and the subscription before it saves them. When the check passes, you see "Config group created successfully."
If the save fails
"Could not parse the file as JSON." appears as soon as you select a file that is not JSON. The other messages appear when you save; they start with "credential verification failed:" and the config group is not saved.
| Message | What to check |
|---|---|
| Could not parse the file as JSON. | The file you selected is not the JSON key file. |
| Invalid JSON Key format | The file is JSON but not a service account key. Download a new key of type JSON. |
| Invalid JSON Key. GCP rejected authentication. | The key was deleted, or its service account was disabled or deleted. |
| GCP Project ... was not found / Invalid Project ID | The project ID is wrong. Use the project ID, not the project name. |
| Cannot connect to GCP Pub/Sub | The key and project do not work together, or the UTMStack server cannot reach Google Cloud on port 443. |
| The service account lacks Pub/Sub Subscriber permission in project ... | The service account cannot read the subscription. Grant Pub/Sub Editor, or Pub/Sub Subscriber plus Pub/Sub Viewer (see Step 3). |
| Subscription ... not found in project ... | The subscription name is wrong, or it is in another project. Enter the subscription ID only, not the full projects/.../subscriptions/... path. |
Check that events arrive
- 1
Check that the subscription receives messages
In the Google Cloud console, open Pub/Sub > Subscriptions and select your subscription. The Metrics tab shows unacknowledged messages. A growing number means the sink publishes but nothing reads yet; a number that drops to zero after you save means UTMStack is reading.
- 2
Open the logs in UTMStack
Open the Gcp card again and click View logs. Log Explorer opens with the
googledata type for the last 24 hours. UTMStack reads the subscription continuously, so events usually appear within a few minutes of being logged. - 3
Check the data source
Go to Configure > Data Sources and open the Integrations tab. Your config group is listed by its name. The row is created when you save, and turns Live after the first events are counted, which can take up to 20 minutes.
Troubleshooting
Errors that happen after the config group is saved, for example a deleted key, do not appear in the web interface. They are written to the logs of the UTMStack event processor on the server, where the integration runs.
| Symptom | Likely cause | Fix |
|---|---|---|
| The subscription has no messages | The sink does not publish to the topic, or its filter matches nothing. | In Logs Router, check the sink's destination and filter. For a topic in another project, grant the sink's writer identity Pub/Sub Publisher on the topic. |
Messages pile up in the subscription and no google events arrive | UTMStack cannot read: the key was deleted or the role was removed after saving. | Create a new key, then edit the config group and choose it with Replace file. Check the service account's roles. |
| Only some activity arrives | Data Access audit logs are off, or the sink filter excludes them. | Turn on Data Access audit logs in IAM & Admin > Audit Logs, and widen the sink filter. |
| Events from another project are missing | One sink and subscription cover one project. | Create a sink in each project to the same topic, or add one config group per project. |
| Events from before you saved are missing | A subscription only receives messages published after it was created. | Search older activity in Logs Explorer in the Google Cloud console. |
| The Integrations row shows Offline although events arrive | The data source's "last seen" time is updated from the ingestion counts, which can take up to about 20 minutes. | Wait 20 minutes and click Refresh. |
Manage config groups
Add more connections: save one config group per project or subscription.
Edit: click the pencil icon next to a config group. The name cannot be changed. The stored key is kept unless you click Replace file and choose a new one. Click Update config group.
Delete: click the bin icon. The config group is deleted at once, without a confirmation, and collection stops. The data source it created stays on the Data Sources page; delete it there if you no longer need it.