Google Cloud

The Google Cloud integration brings your Google Cloud project logs into UTMStack. A Logs Router sink streams the logs into a Pub/Sub topic, and UTMStack pulls them from a subscription on that topic with a service account key. Nothing is installed in your Google Cloud project.

How it works

Pub/Sub is Google Cloud's messaging service: a topic receives messages, and a subscription keeps them until a reader collects and acknowledges them.

  • Cloud Logging sends your project logs through a Logs Router sink to a Pub/Sub topic.

  • UTMStack reads a Pull subscription on that topic, using the JSON key of a service account, and acknowledges each message once it has queued it. Messages that reached the subscription while UTMStack was not reading are collected when it reconnects, as long as Pub/Sub still keeps them.

  • Events are stored with the data type google and parsed by the shipped pipeline google/gcp.yaml (version 2.3.1). UTMStack v12 ships 43 detection rules for this data type; see Google Cloud in the Rules tab.

Before you start

  • In Google Cloud, you need permission to create Pub/Sub topics and subscriptions, Logs Router sinks and service accounts in the project.

  • In UTMStack, you need the Administrator role (the integrations.write permission). Configuration is saved in the tenant you are working in.

  • The UTMStack server must be able to reach Google Cloud over HTTPS (outbound port 443).

Step 1: Create the Pub/Sub topic and subscription

  1. 1

    Create a Pub/Sub topic

    Go to the Pub/Sub topics page in the Google Cloud console and click Create a topic. In Topic ID, enter a unique topic name and save. The Logs Router sink will publish your project logs to this topic.

  2. 2

    Create a subscription

    Open the menu of the topic you just created and click Create subscription. UTMStack pulls messages from this subscription.

    Google Cloud
  3. 3

    Configure the subscription

    Type a name for the subscription. Leave Delivery type as Pull and Expiration as Never expire, then click Create. The default acknowledgement deadline (10 seconds) and message retention are fine.

    Google Cloud

Step 2: Route your logs to the topic

  1. 1

    Open the Logs Router

    Go to Logging > Logs Router. You can find it quickly with the search box.

    Google Cloud
  2. 2

    Create a sink

    Click Create sink and give the sink a name and a description.

    Google Cloud
  3. 3

    Point the sink at the topic

    In Sink destination, select Cloud Pub/Sub topic and choose the topic you created in Step 1. Click Next.

    Google Cloud
  4. 4

    Choose which logs to send (optional)

    Use inclusion and exclusion filters to choose which log entries are sent. If you do not set a filter, all logs of the project are sent to the topic. Click Create sink.

    For example, this inclusion filter sends only Cloud Audit Logs, which most Google Cloud rules read:

    logName:"cloudaudit.googleapis.com"

When you create the sink in the console and the topic is in the same project, Google gives the sink's writer identity permission to publish to the topic. If the topic is in another project, grant the sink's writer identity (shown on the sink's details) the Pub/Sub Publisher role on the topic.

Admin Activity audit logs are always written. Data Access audit logs are off by default for most Google Cloud services; turn them on in IAM & Admin > Audit Logs if you want them in UTMStack.

Step 3: Create a service account and key

  1. 1

    Create a service account

    Go to the service accounts page, select the project that holds your topic and sink, and click Create service account. UTMStack uses this account to pull logs from Pub/Sub.

    Google Cloud
  2. 2

    Grant the Pub/Sub role

    Fill in the form and give the service account the Pub/Sub Editor role so UTMStack can pull from the subscription.

    When you save, UTMStack checks that the subscription exists, which needs permission to read the subscription's details. The Pub/Sub Subscriber role alone does not include that permission, so the check fails with it. If you do not want to grant Pub/Sub Editor on the whole project, grant Pub/Sub Subscriber and Pub/Sub Viewer on the subscription instead.

  3. 3

    Create a JSON key

    Open the service account, go to the Keys tab and add a new key.

    Google Cloud
  4. 4

    Download the key

    Choose JSON as the key type and save the file that Google downloads. Its contents go into UTMStack in Step 4.

    Google Cloud

    Google lets you download the key only once. Keep the file in a safe place.

Step 4: Connect Google Cloud in UTMStack

Each project connection is a config group. You can monitor more than one project by saving more config groups.

  1. 1

    Open the integration

    Go to Configure > Integrations, search for GCP and open the Gcp card. The form is at the end of the setup guide, under Pub/Sub configuration.

    Google Cloud setup guide in UTMStack, showing How it works and the first steps: create a Pub/Sub topic and a subscription
  2. 2

    Fill in the form

    FieldWhat to enter
    Config group nameA name for this connection, for example acme-prod. It cannot be changed later, and it becomes the data source name on the Data Sources page.
    Project IDThe ID of the Google Cloud project that holds your topic, subscription and sink.
    Subscription nameThe Pull subscription you created in Step 1.
    Service account JSON keyClick Select file and choose the JSON key file from Step 3.

    All fields are required. UTMStack does not need the topic name.

  3. 3

    Save

    Click Add config group. UTMStack checks the key, the project and the subscription before it saves them. When the check passes, you see "Config group created successfully."

If the save fails

"Could not parse the file as JSON." appears as soon as you select a file that is not JSON. The other messages appear when you save; they start with "credential verification failed:" and the config group is not saved.

MessageWhat to check
Could not parse the file as JSON.The file you selected is not the JSON key file.
Invalid JSON Key formatThe file is JSON but not a service account key. Download a new key of type JSON.
Invalid JSON Key. GCP rejected authentication.The key was deleted, or its service account was disabled or deleted.
GCP Project ... was not found / Invalid Project IDThe project ID is wrong. Use the project ID, not the project name.
Cannot connect to GCP Pub/SubThe key and project do not work together, or the UTMStack server cannot reach Google Cloud on port 443.
The service account lacks Pub/Sub Subscriber permission in project ...The service account cannot read the subscription. Grant Pub/Sub Editor, or Pub/Sub Subscriber plus Pub/Sub Viewer (see Step 3).
Subscription ... not found in project ...The subscription name is wrong, or it is in another project. Enter the subscription ID only, not the full projects/.../subscriptions/... path.

Check that events arrive

  1. 1

    Check that the subscription receives messages

    In the Google Cloud console, open Pub/Sub > Subscriptions and select your subscription. The Metrics tab shows unacknowledged messages. A growing number means the sink publishes but nothing reads yet; a number that drops to zero after you save means UTMStack is reading.

  2. 2

    Open the logs in UTMStack

    Open the Gcp card again and click View logs. Log Explorer opens with the google data type for the last 24 hours. UTMStack reads the subscription continuously, so events usually appear within a few minutes of being logged.

  3. 3

    Check the data source

    Go to Configure > Data Sources and open the Integrations tab. Your config group is listed by its name. The row is created when you save, and turns Live after the first events are counted, which can take up to 20 minutes.

Troubleshooting

Errors that happen after the config group is saved, for example a deleted key, do not appear in the web interface. They are written to the logs of the UTMStack event processor on the server, where the integration runs.

SymptomLikely causeFix
The subscription has no messagesThe sink does not publish to the topic, or its filter matches nothing.In Logs Router, check the sink's destination and filter. For a topic in another project, grant the sink's writer identity Pub/Sub Publisher on the topic.
Messages pile up in the subscription and no google events arriveUTMStack cannot read: the key was deleted or the role was removed after saving.Create a new key, then edit the config group and choose it with Replace file. Check the service account's roles.
Only some activity arrivesData Access audit logs are off, or the sink filter excludes them.Turn on Data Access audit logs in IAM & Admin > Audit Logs, and widen the sink filter.
Events from another project are missingOne sink and subscription cover one project.Create a sink in each project to the same topic, or add one config group per project.
Events from before you saved are missingA subscription only receives messages published after it was created.Search older activity in Logs Explorer in the Google Cloud console.
The Integrations row shows Offline although events arriveThe data source's "last seen" time is updated from the ingestion counts, which can take up to about 20 minutes.Wait 20 minutes and click Refresh.

Manage config groups

  • Add more connections: save one config group per project or subscription.

  • Edit: click the pencil icon next to a config group. The name cannot be changed. The stored key is kept unless you click Replace file and choose a new one. Click Update config group.

  • Delete: click the bin icon. The config group is deleted at once, without a confirmation, and collection stops. The data source it created stays on the Data Sources page; delete it there if you no longer need it.