Alert management

Alerts are notifications generated by the Wazuh manager after processing events received from Wazuh agents and agentless devices. By default, the alerts are stored in the /var/ossec/logs/alerts/alerts.log and /var/ossec/logs/alerts/alerts.json files.

By default, the Wazuh server uses Filebeat to forward generated alerts to the Wazuh indexer for indexing. Additionally, you can configure the Wazuh manager to forward alerts to other systems involving syslog servers, email systems, and databases.

Alert threshold

An alert threshold is the minimum severity level that must be exceeded for an alert to be triggered. The Wazuh manager assigns a severity level to each event from monitored endpoints based on the matching rule from the ruleset. By default, it only triggers alerts with a severity level of 3 or higher.

Configuration

The alert threshold is configured in the /var/ossec/etc/ossec.conf configuration file on the Wazuh server within the <alerts> XML tag.

The code block below shows the default alert threshold configuration for events and forwarding alerts via email:

<ossec_config>
  <alerts>
    <log_alert_level>3</log_alert_level>
    <email_alert_level>12</email_alert_level>
  </alerts>
</ossec_config>

Where:

  • The <log_alert_level> tag sets the minimum severity level to trigger alerts stored in the /var/ossec/logs/alerts/alerts.log and/or the /var/ossec/logs/alerts/alerts.json file. The default value is 3. The allowed value is any integer from 1 to 16 as referenced in the rules classification guide.

  • The <email_alert_level> tag sets the minimum severity level for an alert to generate an email notification. The default value is 12. The allowed value is any integer from 1 to 16. This setting overrides granular email alert configuration. However, the alert_by_email option within individual rules can override both global and granular alert level thresholds to trigger an email alert.

For details on configuring an alert threshold, refer to the alerts reference guide.

Note

Restart the Wazuh manager when you make any changes to the configuration file. This action ensures that the changes take effect.

Restart the Wazuh manager via the command line interface with the following command:

# systemctl restart wazuh-manager

Forwarding alerts

The Wazuh manager forwards alerts to the Wazuh indexer by default for indexing and analytics capabilities. Additionally, the Wazuh manager provides the capability of configuring and forwarding alerts to other systems for analysis and backup.

Configuring syslog output

You can configure the Wazuh server to send alerts to a syslog server using the syslog_output option. Forwarding alerts to a syslog server can be useful for centralized monitoring and custom reporting.

Configuration

Syslog output is configured in the Wazuh server /var/ossec/etc/ossec.conf configuration file within the <ossec_config> block. By default, the Wazuh manager forwards alerts to syslog servers using port 514 via the UDP protocol.

The code block below shows a sample configuration to forward alerts to a syslog server:

<ossec_config>
  <syslog_output>
    <level>9</level>
    <server>192.168.1.241</server>
  </syslog_output>
</ossec_config>

The configuration options are defined as follows:

  • The <level> tag sets the minimum severity level of the alerts to be forwarded to the syslog server. The example value 9 indicates the Wazuh server only forwards alerts to the syslog server if the alert level is higher than 9. If this option is not defined, the Wazuh server forwards all alerts to the syslog server.

  • The <server> tag sets the IP address or hostname of the syslog server to forward the alerts. The IP address 192.168.1.241 in the configuration is used as an example.

You can learn more about the available configuration options in the syslog_output reference guide.

Restart the Wazuh manager service to apply the changes after every configuration:

# systemctl restart wazuh-manager

You can also forward alerts to multiple syslog servers by defining multiple <syslog_output> blocks within the <ossec_config> block in the /var/ossec/etc/ossec.conf configuration file.

<ossec_config>
  <syslog_output>
    <server>192.168.1.240</server>
  </syslog_output>

  <syslog_output>
    <level>9</level>
    <server>192.168.1.241</server>
  </syslog_output>
</ossec_config>

In the configuration above,

  • The first <syslog_output> block sends all alerts without filtering to the syslog server with IP address 192.168.1.240.

  • The second <syslog_output> block sends alerts to the syslog server 192.168.1.241 only if the alert level is higher than 9.

Configuring email alerts

Wazuh provides a feature to send alerts to email systems when they are generated on a Wazuh server. You can configure it to send email alerts to one or more email addresses when rules are triggered or based on customized settings. This configuration can help you for daily event reports and more.

A sample email sent by Wazuh when the rule ID 553 is triggered is shown below:

  Wazuh Notification.
  2024 Apr 29 08:58:30

  Received From: wazuh-server->syscheck
  Rule: 553 fired (level 7) -> "File deleted."
  Portion of the log(s):

  File '/var/ossec/test_dir/somefile.
  txt' deleted
  Mode: realtime

  Attributes:
   - Size: 0
   - Permissions: rw-r--r--
   - Date: Mon Apr 29 08:46:12 2024
   - Inode: 841858
   - User: root (0)
   - Group: root (0)
   - MD5: d41d8cd98f00b204e9800998ecf8427e
   - SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
   - SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855


--END OF NOTIFICATION

Generic email options

To configure Wazuh to send email alerts, we configure the email options within the <global> section of the /var/ossec/etc/ossec.conf file.

A sample email configuration to send alerts to the email address me@test.com is shown below:

<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <email_to>me@test.com</email_to>
    <smtp_server>mail.test.com</smtp_server>
    <email_from>wazuh@test.com</email_from>
  </global>
  ...
</ossec_config>

Once the above has been configured, the email_alert_level option needs to be set to the minimum alert level to trigger an email. By default, this level is set to 12.

This example configuration below sets the minimum level to send email alerts to 10:

<ossec_config>
  <alerts>
    <email_alert_level>10</email_alert_level>
  </alerts>
  ...
</ossec_config>

For more information about the available email configuration options, see the global and alerts reference guides.

Restart the Wazuh manager service to apply the changes after every configuration:

# systemctl restart wazuh-manager

Warning

Wazuh doesn't handle SMTP authentication. If your email service uses this, you need to configure a server relay.

Granular email options

Wazuh allows granular configuration options for email alerts. This setting extends the generic email options configured in the <global> section of the /var/ossec/etc/ossec.conf file. Granular email configurations are defined within the <email_alerts> tag of the /var/ossec/etc/ossec.conf file.

Below are some sample granular configurations for sending alerts via email. For more information, see the email_alerts section.

Warning

The minimum severity level configured in the <alerts> section applies to and overrides these granular email configurations. For example, if you configure the Wazuh manager to send an email when rule 526 is triggered, but the rule has a level lower than the minimum level specified in the <alerts> section, the alert will not be sent.

Email alert based on level

This option configures the Wazuh manager to send email alerts when the severity level is equal to or greater than the set value. This option is configured as follows:

<email_alerts>
  <email_to>you@example.com</email_to>
  <level>4</level>
  <do_not_delay/>
</email_alerts>

This configuration allows the Wazuh manager to send an email to you@example.com when any rule with a level greater than or equal to 4 is triggered.

Note

If the severity level here is less than the email_alert_level configured in the <alerts> section, the email will not be sent.

Email alert based on the event location

The event_location option involves sending email alerts based on the location from where the event originated. The generated alert must match the event location to be forwarded via email. The allowed values for this option are the Wazuh agent name, hostname, IP address, or log file.

This option is configured as follows:

<email_alerts>
  <email_to>you@example.com</email_to>
  <event_location>server1</event_location>
  <do_not_delay/>
</email_alerts>

This configuration allows the Wazuh manager to send an email to you@example.com when the events that generated the alerts originated on the Wazuh agent with name server1.

Learn more about the event_location option in the reference guide.

Email based on rules ID

The rule_id option is used to send alert emails based on rule IDs. This option limits the sending of emails only when specific defined rules are tripped.

This option is configured as follows:

<email_alerts>
  <email_to>you@example.com</email_to>
  <rule_id>515, 516</rule_id>
  <do_not_delay/>
</email_alerts>

This configuration allows the Wazuh manager to send an email to you@example.com when the rules 515 or 516 are triggered.

Learn more about the rule_id option in the reference guide.

Email based on the rule group

The group option can be configured to send an email based on one or more rule groups that alerts belong to.

This option is configured as follows:

<email_alerts>
  <email_to>you@example.com</email_to>
  <group>pci_dss_10.6.1,</group>
</email_alerts>

This configuration allows the Wazuh manager to send an email to you@example.com when any rule that is part of the pci_dss_10.6.1 group is triggered on any Wazuh monitored endpoint.

Learn more about the group option in the reference guide.

Multiple options and multiple emails

Email alerts can be sent to multiple email addresses, each one with unique criteria.

The example configuration below shows how to send email alerts with multiple criteria to multiple email addresses:

<ossec_config>
  <email_alerts>
    <email_to>alice@test.com</email_to>
    <event_location>endpoint1|endpoint2</event_location>
  </email_alerts>

  <email_alerts>
    <email_to>is@test.com</email_to>
    <event_location>/log/secure$</event_location>
  </email_alerts>

  <email_alerts>
    <email_to>bob@test.com</email_to>
    <event_location>192.168.</event_location>
  </email_alerts>

  <email_alerts>
    <email_to>david@test.com</email_to>
    <level>12</level>
  </email_alerts>
</ossec_config>

This configuration sends:

  • An email to alice@test.com if any alert on endpoint1 or endpoint2 is triggered.

  • An email to is@test.com if the alerts came from the /log/secure file.

  • An email to bob@test.com if the alerts came from any endpoint on the 192.168.0.0/24 network.

  • An email to david@test.com if the alerts have a level equal to or higher than 12.

Force forwarding an alert by email

The minimum severity level to send an alert via email is 12 by default. You can configure the Wazuh manager to forcefully send an email alert below the configured minimum severity level. To do so, you need to use one of the rule options below:

  • alert_by_email to always alert by email.

  • no_email_alert to never alert by email.

  • no_log to not log this alert.

For example, the rule definition below sends an email every time rule 502 is triggered, regardless of what the minimum severity level is set to:

<rule id="502" level="3">
  <if_sid>500</if_sid>
  <options>alert_by_email</options>
  <match>Ossec started</match>
  <description>Ossec server started.</description>
</rule>

SMTP server with authentication

Wazuh email alerts do not support SMTP servers with authentication, such as Gmail. However, you can send these emails via a server relay like Postfix.

Perform the steps below on your relay server to configure Postfix with Gmail.

  1. Run this command to install the required packages. Select No configuration, if prompted about the mail server configuration type.

    # yum update && yum install postfix mailx cyrus-sasl cyrus-sasl-plain
    
  2. Append these lines to the /etc/postfix/main.cf file to configure Postfix. Create the file if missing.

    relayhost = [smtp.gmail.com]:587
    smtp_sasl_auth_enable = yes
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
    smtp_sasl_security_options = noanonymous
    smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt
    smtp_use_tls = yes
    
  3. Set the credentials of the sender in the /etc/postfix/sasl_passwd file and create a database file for Postfix. Replace the <USERNAME> and <PASSWORD> variables with sender’s email address username and password respectively.

    # echo [smtp.gmail.com]:587 <USERNAME>@gmail.com:<PASSWORD> > /etc/postfix/sasl_passwd
    # postmap /etc/postfix/sasl_passwd
    

    Note

    The password must be an App Password. App Passwords can only be used with accounts that have 2-Step Verification turned on.

  4. Secure your password DB file so that only the root user has full read and write access to it. This is because the /etc/postfix/sasl_passwd and /etc/postfix/sasl_passwd.db files have plaintext credentials.

    # chown root:root /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
    # chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
    
  5. Restart Postfix to effect the configuration changes:

    # systemctl restart postfix
    
  6. Run the following command to test the configuration:

    # echo "Test mail from postfix" | mail -s "Test Postfix" -r "<CONFIGURED_EMAIL>" <RECEIVER_EMAIL>
    

    Replace:

    • <CONFIGURED_EMAIL> with your configured email address.

    • <RECEIVER_EMAIL> with the email address of the recipient.

    The command sends an email to the receiver’s email with subject Test Postfix and body Test mail from postfix.

    If you get the error message fatal: tls_fprint: error computing md5 message digest in the /var/log/maillog file, run the following commands to switch Postfix from the default MD5 hashing function to SHA-256:

    # postconf -e smtp_tls_fingerprint_digest=sha256
    # postconf -e smtpd_tls_fingerprint_digest=sha256
    
  7. Configure email notifications within the <global> tag of the Wazuh server’s /var/ossec/etc/ossec.conf file as follows:

    <global>
      <email_notification>yes</email_notification>
      <smtp_server>localhost</smtp_server>
      <email_from><USERNAME>@gmail.com</email_from>
      <email_to><RECEIVER_EMAIL></email_to>
    </global>
    

    Where:

    • <email_notification> toggles the use of email alerting.

    • <smtp_server> defines the SMTP server to use to deliver alerts.

    • <email_from> specifies the email address of the configured sender. Replace <USERNAME> with your configured username of your email address.

    • <email_to> specifies the email address of the recipient of alerts. Replace <RECEIVER_EMAIL> with the email address of the recipient.

  8. Restart the Wazuh manager to apply the changes:

    # systemctl restart wazuh-manager
    

Configuring database output

Wazuh supports forwarding alerts to database systems. You can configure the Wazuh manager to output generated alerts into a database. To achieve this configuration, you must compile the Wazuh manager from sources with the database type that you want to use. Wazuh currently supports MySQL and PostgreSQL databases.

Note

This guide assumes you have already installed MySQL or PostgreSQL and know how to create the users and the databases.

Prerequisites

You need to install the development libraries for the database system you want to configure and compile the Wazuh manager to use the required database system.

  1. Install the development libraries for the database system:

    • For MySQL:

      # yum install mysql-devel
      
    • For PostgreSQL:

      # yum install postgresql-devel
      
  2. Install the dependencies as outlined in the installing dependencies section.

  3. Download and extract the latest version of Wazuh:

    # curl -Ls https://github.com/wazuh/wazuh/archive/v4.9.1.tar.gz | tar zx
    
  4. Execute the following commands to switch to the Wazuh directory and specify the database type to use, replacing the <DATABASE_TYPE> variable with either mysql or pgsql:

    # cd wazuh-4.9.1/src
    # make deps && make TARGET=server DATABASE=<DATABASE_TYPE>
    

    Note

    Depending on your system specifications, the compilation process might take some time.

  5. Run the install.sh script. It displays a wizard to guide you through the installation process using the Wazuh sources:

    # cd ..
    # ./install.sh
    
  6. When the script asks what kind of installation you want, type manager to install the Wazuh manager:

    1- What kind of installation do you want (manager, agent, local, hybrid, or help)? manager
    

    Note

    During the installation, you can decide the installation path. Execute the install.sh and select the language, set the installation mode to manager, then set the installation path (Choose where to install Wazuh [/var/ossec]). The default path of installation is /var/ossec. A commonly used custom path might be /opt.

    Warning

    Be extremely careful not to select a critical installation directory if you choose a different path than the default. If the directory already exists, the installer will ask to delete the directory or proceed by installing Wazuh inside it.

  7. The installer asks if you want to start Wazuh at the end of the installation. If you choose not to, you can start it later with the command below:

    # systemctl restart wazuh-manager
    

Database configuration

According to your database system, create a new database, set up the database user, and add the schema located in the src/os_dbd directory of the source code with the following commands:

  • For MySQL:

    # mysql -u root -p
    
    mysql> CREATE DATABASE Alerts_DB;
    Query OK, 1 row affected (2.34 sec)
    
    mysql> CREATE USER '<DATABASE_USER>'@'<DATABASE_SERVER_IP>' IDENTIFIED BY '<DATABASE_USER_PASSWORD>';
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> GRANT INSERT,SELECT,UPDATE,CREATE,DELETE,EXECUTE on Alerts_DB.* to '<DATABASE_USER>'@'<DATABASE_SERVER_IP>';
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> FLUSH PRIVILEGES;
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> quit;
    

    Replace the following variables in the commands above:

    • <DATABASE_USER> with the user you want to create for the database server.

    • <DATABASE_SERVER_IP> with the IP address of the database server.

    • <DATABASE_USER_PASSWORD> with the user password to access the database server.

    # mysql -u root -p Alerts_DB < src/os_dbd/mysql.schema
    
  • For PostgreSQL:

    # sudo -u postgres createuser -P <DATABASE_USER>
    # sudo -u postgres createdb -O <DATABASE_USER> Alerts_DB
    # psql -U <DATABASE_USER> -d Alerts_DB -f src/os_dbd/postgresql.schema
    

    Replace <DATABASE_USER> with the user you want to create for the database server.

Note

You will be prompted twice to provide a password when creating the user. Note this password as it is required when configuring the Wazuh manager.

Wazuh manager configuration

Perform the following steps to configure the Wazuh manager to send alerts and other data to the database system.

  1. Add the following block of code within the <ossec_config> block of the /var/ossec/etc/ossec.conf file on the Wazuh server:

    • For MySQL:

      <database_output>
        <hostname><DATABASE_SERVER_IP></hostname>
        <username><DATABASE_USER></username>
        <password><DATABASE_USER_PASSWORD></password>
        <database>Alerts_DB</database>
        <type>mysql</type>
      </database_output>
      
    • For PostgreSQL:

      <database_output>
        <hostname><DATABASE_SERVER_IP></hostname>
        <username><DATABASE_USER></username>
        <password><DATABASE_USER_PASSWORD></password>
        <database>Alerts_DB</database>
        <type>postgresql</type>
      </database_output>
      

    Where:

    • <hostname> specifies the IP address of the database server. Replace <DATABASE_SERVER_IP> the IP address of the database server.

    • <username> specifies the user to access the database. Replace <DATABASE_USER> with the database user created above.

    • <password> specifies the user password to access the database. Replace <DATABASE_USER_PASSWORD> with the user password created above.

    • <database> specifies the name of the database in which to store the alerts. For example, Alerts_DB as specified in the configuration above.

    • <type> specifies the type of database (MySQL or PostgreSQL). The allowed values are mysql or pgsql.

    For more information on the database_output option, see the database_output section of the reference guide.

  2. Restart the Wazuh manager service to apply the changes:

    # systemctl restart wazuh-manager
    
  3. Run the command below to verify that the Wazuh manager is connected to the database:

    # grep wazuh-dbd /var/ossec/logs/ossec.log
    
    2024/06/24 14:49:11 wazuh-dbd: INFO: Connected to database 'Alerts_DB' at '127.0.0.1'.
    

The database will now start receiving data from the Wazuh manager.