Monitoring Google Cloud Storage buckets

Setting up log delivery to a Google Cloud Storage bucket

You need to create Storage buckets and enable logging before logs can be delivered to them. The log delivery for any bucket must be set up using the Google Cloud Shell, gsutil tool, the XML API, or the JSON API. Follow the usage logs and storage logs documentation for the most up-to-date instructions on enabling this feature.

In the following steps, we use the Google Cloud Shell to create a bucket and enable logging.

1. On Google Cloud Console, click the Shell button to activate Cloud Shell and authenticate your Google Cloud SDK.

   

2. Create a bucket to store your logs using the following command:

   $ gcloud storage buckets create gs://<YOUR_BUCKET_NAME>
   

   Replace <YOUR_BUCKET_NAME> with your bucket name.

   Make sure to follow the Google naming guidelines to name your bucket.

3. Grant Google Cloud Storage the roles/storage.objectCreator role to the bucket:

   $ gcloud storage buckets add-iam-policy-binding gs://<YOUR_BUCKET_NAME> --member=group:cloud-storage-analytics@google.com --role=roles/storage.objectCreator
   

4. Enable logging for your bucket using the --log-bucket flag:

   $ gcloud storage buckets update  gs://<YOUR_BUCKET_NAME> --log-bucket=gs://<YOUR_BUCKET_NAME>
   

5. Check logging status:

   $ gcloud storage buckets describe gs://<YOUR_BUCKET_NAME> --format="default(logging_config)"
   

If logging is enabled, the server returns the logging configuration in the response:

logging_config:
  logBucket: <YOUR_BUCKET_NAME>
  logObjectPrefix: <YOUR_BUCKET_NAME>

If logging is not enabled, the following is returned:

null

Configuring multiple buckets

You can configure the Wazuh server or agent to pull logs from multiple buckets. To do so, you must add multiple <bucket> blocks within the <gcp-bucket> section of the Wazuh server or agent configuration file.

Find below an example configuration for multiple buckets:

<gcp-bucket>
   <run_on_start>yes</run_on_start>
   <interval>1m</interval>
   <bucket type="access_logs">
       <name>wazuh-test-bucket</name>
       <credentials_file>/var/ossec/wodles/gcloud/wazuh-test-bucket-credentials.json</credentials_file>
   </bucket>

   <bucket type="access_logs">
       <name>wazuh-test-bucket-2</name>
       <credentials_file>/var/ossec/wodles/gcloud/wazuh-test-bucket2-credentials.json</credentials_file>
       <only_logs_after>2021-JUN-01</only_logs_after>
       <path>access_logs/</path>
   </bucket>

   <bucket type="access_logs">
       <name>wazuh-test-bucket-3</name>
       <credentials_file>/var/ossec/wodles/gcloud/wazuh-test-bucket3-credentials.json</credentials_file>
       <path>access_logs</path>
       <remove_from_bucket>no</remove_from_bucket>
   </bucket>

 </gcp-bucket>

Where:

• <name>:Contains the name of the Google Cloud Storage bucket from which logs are read.

• <credentials_file>:Contains the path to the Google Cloud credentials file.

• <only_logs_after>: Parses logs from a specific date onwards. It must follow the YYYY-MM-DD format.

• <remove_from_bucket>: Sets whether the logs should be removed from the Google Cloud Storage bucket once they are read. The possible values are no and yes.

Restart the Wazuh manager or agent service to apply the changes:

# systemctl restart wazuh-manager

# systemctl restart wazuh-agent

First execution

If no only_logs_after value is provided, the module will only fetch the logs of the date of the execution.

Creation time in Google Cloud Storage bucket contents

When using the only_logs_after tag, the Wazuh module checks the creation time of each item in the Google Cloud Storage bucket to determine if a file should be processed or not. This means that if the user manually moves any blob inside the specified bucket, its creation date changes, and the Wazuh module for Google Cloud Storage processes it again as it is considered a new blob.

Any date in the file's name is ignored, and only the creation date is used to determine whether or not a file should be processed.

Older logs

The Wazuh module for Cloud Storage buckets only looks for new logs in buckets based on the key of the last processed log object, which includes the datetime stamp. When the only_logs_after option date is set to a datetime earlier than previous module executions, the system will ignore older logs.

Logging level

The Google Cloud integration uses the wazuh_modules.debug level to set its verbosity level. This switches between different logging levels for debugging and troubleshooting purposes.

Reparse

Using the reparse option will re-fetch all Google Cloud Storage bucket logs from the starting date of the integration until the present.

To fetch and process older logs, manually run the /var/ossec/wodles/gcloud/gcloud tool using the --reparse option.

# /var/ossec/wodles/gcloud/gcloud --integration_type access_logs --bucket_name 'wazuh-example-bucket' --credentials_file credentials.json --reparse --only_logs_after '2021-Jun-10' --log_level 2

Where:

• --only_logs_after parameter sets the time for the starting point. If you don't provide an only_logs_after value, the module uses the date of the first file processed.

• --log_level 2 parameter gets a verbose output. This is useful to show that the script works, especially when handling a large amount of data.