Troubleshooting

The below information is intended to assist in troubleshooting issues.

Checking if the Wazuh module for AWS is running

When the Wazuh module for AWS runs it writes its output in the ossec.log file. This log file can be found in /var/ossec/logs/ossec.log and under Server Management > Logs if you use the Wazuh dashboard. It is possible to check if the Wazuh module for AWS is running without issues by looking in the /var/ossec/logs/ossec.log file. These are the messages that are displayed in the file depending on how the Wazuh module for AWS has been configured:

  • When the Wazuh module for AWS is starting:

    2022/03/04 00:00:00 wazuh-modulesd:aws-s3: INFO: Module AWS started
2022/03/04 00:00:00 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.

  • When Scheduled scan is set:

    2022/03/04 00:00:00 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.
2022/03/04 00:00:00 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

  • When the Wazuh module for AWS has finished its execution and is waiting until the interval condition is met:

    2022/03/04 00:00:00 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

Enabling debug mode

It is possible to obtain additional information about the Wazuh module for AWS execution by enabling the debug mode. This is used to see INFO or DEBUG messages. There are three different debug levels available:

  • Debug level 0: Only ERROR and WARNING messages are written in the /var/ossec/logs/ossec.log file. This is the default value.

  • Debug level 1: In addition to ERROR and WARNING messages, INFO messages are written in the /var/ossec/logs/ossec.log file too. They are useful to check the execution of the module without having to manage large amounts of DEBUG messages.

  • Debug level 2: This is the highest level of verbosity. Every message type is dump into the /var/ossec/logs/ossec.log file, including DEBUG messages which contain the details of the different operations performed by the Wazuh module. This is the recommended mode when troubleshooting the Wazuh module for AWS.

Follow these steps to enable debug mode:

  1. Add the following line to the /var/ossec/etc/local_internal_options.conf file of the Wazuh server or agent, specifying the desired debug level:

    wazuh_modules.debug=2

  2. Restart the Wazuh service.

    Wazuh manager

    # systemctl restart wazuh-manager

    Wazuh agent

    # systemctl restart wazuh-agent

Note

Don't forget to disable debug mode once the troubleshooting has finished. Leaving debug mode enabled could result in the addition of large amounts of logs in the /var/ossec/logs/ossec.log file.

Checking if logs are being processed

The easiest way to check if the logs are being processed, regardless of the type of bucket or service configured and regardless of whether alerts are being generated or not is by using the logall_json parameter.

To understand how the logall_json parameter works it is necessary to learn about the flow that is followed when processing a log until the corresponding alert is displayed on the Wazuh dashboard. It is as follows:

  1. The Wazuh module for AWS downloads the logs available in AWS for the requested date and path. Check the Considerations for the Wazuh module for AWS configuration page to learn more about how to properly filter the logs.

  2. The content of these logs is sent to the analysis engine in the form of an Event.

  3. The analysis engine evaluates these events and compares them with the different rules available. If the event matches any of the rules an alert is generated, which is what ultimately is shown on the Wazuh dashboard.

With this in mind, it is possible to enable the Wazuh archives using the logall_json option. When this option is activated, Wazuh stores into the /var/ossec/logs/archives/archives.json file every event sent to the analysis engine whether they tripped a rule or not. By checking this file, it is possible to determine if the AWS events are being sent to the analysis engine and therefore the Wazuh module for AWS is working as expected.

Note

Don't forget to disable the logall_json parameter once the troubleshooting has finished. Leaving it enabled could result in high disk space consumption.

Common problems and solutions

Unable to locate credentials

The Wazuh module for AWS does not work and the following error messages appear in the /var/ossec/logs/ossec.log file:

2022/03/03 16:01:48 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Returned exit code 12
2022/03/03 16:01:48 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Unable to locate credentials

Solution

No authentication method was provided within the configuration of the Wazuh module for AWS. Check the Configuring AWS credentials section to learn more about the different options available and how to configure them.

Invalid credentials to access S3 Bucket

The Wazuh module for AWS does not work and the following error messages appear in the /var/ossec/logs/ossec.log file:

2022/03/03 16:06:56 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Returned exit code 3
2022/03/03 16:06:56 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Invalid credentials to access S3 Bucket

Solution

Make sure the credentials provided grant access to the requested S3 bucket and the bucket itself exists.

The config profile could not be found

The Wazuh module for AWS does not work and the following error messages appear in the /var/ossec/logs/ossec.log file:

2022/03/03 15:49:34 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Returned exit code 12
2022/03/03 15:49:34 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  The config profile (default) could not be found

Solution

Ensure the profile value specified in the configuration matches an existing one placed in /root/.aws/credentials. Check the Profiles section to learn more about configuring a profile for authentication.

The security token included in the request is invalid

The Wazuh module for AWS does not work and the following error messages appear in the /var/ossec/logs/ossec.log file:

2022/03/03 16:16:18 wazuh-modulesd:aws-s3: WARNING: Service: cloudwatchlogs  -  Returned exit code 12
2022/03/03 16:16:18 wazuh-modulesd:aws-s3: WARNING: Service: cloudwatchlogs  -  An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

Solution

No credentials were provided to attempt to access to CloudWatch Logs or that the credentials provided don't grant access to CloudWatch Logs. Check the Configuring AWS credentials section to learn more about the different options available and how to configure them.

There are no AWS alerts present on the Wazuh dashboard

The Wazuh module for AWS is running but no alerts are displayed on the Wazuh dashboard.

Solution

First of all, review ERROR or WARNING messages in the /var/ossec/logs/ossec.log file by enabling debug mode. If the Wazuh module for AWS is running as expected but no alerts are being generated it could mean there is no reason for alerts to be raised in first place. Check the following to verify this:

  • Make sure there is data available for the given date.

    When running, the Wazuh module for AWS requests AWS for the logs corresponding to the date indicated using the only_logs_after parameter. If this parameter is not specified, it will try to obtain the logs corresponding to the day of execution. Make sure you are specifying a value for only_logs_after and that there is data available for that particular date. Check the Considerations for the Wazuh module for AWS configuration page to learn more about how to properly filter the logs using the only_logs_after parameter.

  • Check if the events are being sent to the analysis engine.

    A common scenario is that no alerts are being generated because the events do not match any of the available rules. Take a look at the Checking if logs are being processed section to learn how to check if the AWS logs are being sent to the analysis engine.

CloudWatch Logs integration is running but no alert is shown on the Wazuh dashboard

The Wazuh module for AWS is running without any error or warning messages, but no alerts from CloudWatch Logs are displayed on the Wazuh dashboard.

Solution

A common scenario is that no alerts are being generated because the events do not match any of the available rules. Take a look at the Checking if logs are being processed section to learn how to check if the AWS logs are being sent to the analysis engine.

Take into account that Wazuh does not provide default rules for the different logs that can be found in CloudWatch Logs, since they can have any type of format and come from any source. Because of this, if a user wants to make use of this integration to process any custom log they will most likely have to configure their own rules for them. Take a look at the Custom rules section to learn more about this topic.

Interval overtaken message is present in the log file

The Interval overtaken message is present in the /var/ossec/logs/ossec.log file.

Solution

Not an issue but a warning. This means the time the Wazuh module for AWS required to finish the last execution was greater than the interval value defined. It is important to note that the next run will not start until the previous one is finished.

Error codes reference

  1. Errors in the /var/ossec/logs/ossec.log file of the Wazuh server or agent.

    The exit codes and their possible remediation are as follows:

    Code

    Description

    Possible remediation

    1

    Unknown error

    Programming error. Please, open an issue in the Wazuh GitHub repository with the trace of the error.

    2

    SIGINT

    The module stopped due to an interrupt signal.

    3

    Invalid credentials to access S3 bucket

    Make sure that your credentials are correct. For more information, see the Configuring AWS credentials section.

    4

    boto3 module missing

    Install boto3 library. For more information, see the Installing dependencies section.

    5

    Unexpected error accessing SQLite DB

    Check that no more instances of the Wazuh module for AWS are running at the same time.

    6

    Unable to create SQLite DB

    Make sure that the wodle has the right permissions in its directory.

    7

    Unexpected error querying/working with objects in S3

    Check that no more instances of the Wazuh module for AWS are running at the same time.

    8

    Failed to decompress file

    Only .gz and .zip compression formats are supported.

    9

    Failed to parse file

    Ensure that the log file contents have the expected structure.

    10

    pyarrow module missing

    Install pyarrow library. For more information, see the Installing dependencies section.

    11

    Unable to connect to Wazuh

    Ensure that Wazuh is running.

    12

    Invalid type of bucket

    Check if the type of bucket is one of the supported.

    13

    Error sending message to Wazuh

    Make sure that Wazuh is running.

    14

    Empty bucket

    Make sure that the path to the log files is correct.

    15

    Invalid VPC endpoint URL

    Ensure that the VPC endpoint URL provided is correct.

    16

    Throttling error

    AWS is receiving more than 10 requests per second. Try to run the module again when the number of requests to AWS has decreased. For more information see the Connection configuration for retries section.

    17

    Invalid file key format

    Ensure that the file path follows the format specified in the supported services

    18

    Invalid prefix

    Make sure that the indicated path exists in the S3 bucket.

    19

    The server datetime and datetime of the AWS environment differ

    Make sure that the server datetime is correctly set.

    20

    Unable to find SQS

    Make sure that the sqs_name value in the Wazuh module for AWS configuration in the ossec.conf file is correct.

    21

    Failed fetch/delete from SQS

    Check that no more instances of the Wazuh module for AWS are running at the same time.

    22

    Invalid region

    Check the provided region in the ossec.conf file.

    23

    Profile not found

    Check the provided aws_profile in the ossec.conf file.