Wazuh and Open Distro for Elasticsearch

This section guides you through the upgrade process of the Wazuh server, Elasticsearch, and Kibana for the Open Distro for Elasticsearch distribution.

Note

You need root user privileges to run all the commands described below.

Preparing the upgrade

  1. Add the Wazuh repository. You can skip this step if the repository is already present and enabled on the node.

    1. Import the GPG key.

      # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH

    2. Add the repository.

      # echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://packages.wazuh.com/4.x/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo

  2. Repeat the previous step for every Wazuh node.

Upgrading the Wazuh manager

When upgrading a multi-node Wazuh manager cluster, run the upgrade in every node to make all the Wazuh manager nodes join the cluster. Start with the master node to reduce server downtime.

Note

Upgrading from Wazuh 4.2.x or lower creates the wazuh operating system user and group to replace ossec. To avoid upgrade conflicts, make sure that the wazuh user and group are not present in your operating system.

  1. Upgrade the Wazuh manager to the latest version.

    # yum upgrade wazuh-manager-4.5.4-1

    Note

    If the /var/ossec/etc/ossec.conf configuration file was modified, it will not be replaced by the upgrade. You will therefore have to add the settings of the new capabilities manually. More information can be found in User manual.

  2. Repeat the previous steps for every Wazuh manager node.

Upgrading Open Distro for Elasticsearch

Preparations

  1. Stop the services:

    # systemctl stop filebeat
# systemctl stop kibana

  2. Prepare the repositories. Wazuh now hosts the Open Distro packages. In order to prevent accidental upgrades, it is recommended to disable the Open Distro repository. Besides, if the Elastic repository was added, it is also recommended to disable it:

    1. Disable the Open Distro for Elasticsearch repository:

      # sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/opendistroforelasticsearch-artifacts.repo

    2. (Optional) Disable the Elastic Stack basic license repository:

      # sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo

    3. Add the Wazuh 4.x repository:

      1. Import the GPG key:

        # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH

      2. Add the repository:

        # cat > /etc/yum.repos.d/wazuh.repo << EOF
[wazuh]
gpgcheck=1
gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
enabled=1
name=EL-\$releasever - Wazuh
baseurl=https://packages.wazuh.com/4.x/yum/
protect=1
EOF

      1. Clean the YUM cache:

      # yum clean all

Upgrade

This guide explains how to perform a rolling upgrade, which allows you to shut down one node at a time for minimal disruption of service. The cluster remains available throughout the process.

The IP address 127.0.0.1 is used in the commands below. If Elasticsearch is bound to a specific IP address, replace 127.0.0.1 with your Elasticsearch IP address. If using http, the option -k must be omitted, and if not using user/password authentication, -u must be omitted.

  1. Disable shard allocation:

    curl -X PUT "https://127.0.0.1:9200/_cluster/settings"  -u <username>:<password> -k -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
'

  2. Stop non-essential indexing and perform a synced flush:

    # curl -X POST "https://127.0.0.1:9200/_flush/synced" -u <username>:<password> -k

  3. Shut down a single node:

    # systemctl stop elasticsearch

  4. Upgrade the node you shut down:

    # yum install opendistroforelasticsearch-1.13.2

  5. Restart the service:

    Warning

    Add the following configuration to mitigate Apache Log4j2 Remote Code Execution (RCE) vulnerability - CVE-2021-44228 - ESA-2021-31.

    # mkdir -p /etc/elasticsearch/jvm.options.d
# echo '-Dlog4j2.formatMsgNoLookups=true' > /etc/elasticsearch/jvm.options.d/disabledlog4j.options
# chmod 2750 /etc/elasticsearch/jvm.options.d/disabledlog4j.options
# chown root:elasticsearch /etc/elasticsearch/jvm.options.d/disabledlog4j.options

    # systemctl daemon-reload
# systemctl enable elasticsearch
# systemctl start elasticsearch

  6. Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

    # curl -X GET "https://127.0.0.1:9200/_cat/nodes" -u <username>:<password> -k

  7. Reenable shard allocation:

    curl -X PUT "https://127.0.0.1:9200/_cluster/settings" -u <username>:<password> -k -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}
'

  8. Before upgrading the next node, wait for the cluster to finish shard allocation:

    # curl -X GET "https://127.0.0.1:9200/_cat/health?v" -u <username>:<password> -k

  9. Repeat the steps for every Elasticsearch node.

Note

The Open Distro for Elasticsearch performance analyzer plugin is installed by default and can have a negative impact on system resources. We recommend removing it with the following command /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro-performance-analyzer.

Upgrading Filebeat

  1. Upgrade Filebeat:

    # yum install filebeat-7.10.2

  2. Download the alerts template for Elasticsearch:

    # curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.5.4/extensions/elasticsearch/7.x/wazuh-template.json
# chmod go+r /etc/filebeat/wazuh-template.json

  3. Download the Wazuh module for Filebeat:

    # curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.2.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module

  4. Edit the /etc/filebeat/filebeat.yml configuration file. This step is only needed for the upgrade of a Distributed installation. In case of having an All-in-one installation, the file is already configured:

    output.elasticsearch:
  hosts: ["<elasticsearch_ip>:9200"]

    Replace <elasticsearch_ip> with the IP address or the hostname of the Elasticsearch server.

    During the installation, the default username and password were used. If those credentials were changed, replace those values in the filebeat.yml configuration file.

  5. Restart Filebeat:

    # systemctl daemon-reload
# systemctl enable filebeat
# systemctl start filebeat

  6. Upload the new Wazuh template to Elasticsearch. This step can be omitted in Wazuh single-node installations:

    # filebeat setup --index-management -E output.logstash.enabled=false

Upgrading Kibana

Warning

The location of the Wazuh Kibana plugin configuration file has been moved to /usr/share/kibana/data/wazuh/config/wazuh.yml

  1. Copy the Wazuh Kibana plugin configuration file to its new location:

    Create the new directory and copy the Wazuh Kibana plugin configuration file.

    # mkdir -p /usr/share/kibana/data/wazuh/config/
# cp /usr/share/kibana/optimize/wazuh/config/wazuh.yml /usr/share/kibana/data/wazuh/config/wazuh.yml

  2. (For upgrades from 3.x versions) Replace the value user by username and set the username and password as wazuh-wui in the file /usr/share/kibana/data/wazuh/config/wazuh.yml:

    hosts:
  - default:
      url: https://localhost
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: false

  3. Remove the Wazuh Kibana plugin:

    # cd /usr/share/kibana/
# sudo -u kibana bin/kibana-plugin remove wazuh

  4. Upgrade Kibana:

    # yum install opendistroforelasticsearch-kibana-1.13.2

  5. (For upgrades from 3.x versions) Remove generated bundles and the wazuh-registry.json file:

    # rm -rf /usr/share/kibana/optimize/bundles
# rm -f /usr/share/kibana/optimize/wazuh/config/wazuh-registry.json

  6. Update file permissions. This will prevent errors when generating new bundles or updating the Wazuh Kibana plugin:

    # chown -R kibana:kibana /usr/share/kibana/data
# chown -R kibana:kibana /usr/share/kibana/plugins

  7. Install the Wazuh Kibana plugin:

    # cd /usr/share/kibana/
# sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/4.x/ui/kibana/wazuh_kibana-4.5.4_7.10.2-1.zip

  8. Update configuration file permissions:

    # sudo chown kibana:kibana /usr/share/kibana/data/wazuh/config/wazuh.yml
# sudo chmod 600 /usr/share/kibana/data/wazuh/config/wazuh.yml

  9. For installations on Kibana 7.6.x version and higher, it is recommended to increase the heap size of Kibana to ensure the Kibana's plugins installation:

    # cat >> /etc/default/kibana << EOF
NODE_OPTIONS="--max_old_space_size=2048"
EOF

  10. Link Kibana’s socket to privileged port 443:

    # setcap 'cap_net_bind_service=+ep' /usr/share/kibana/node/bin/node

  11. (Recommended) Enable multitenancy in the /etc/kibana/kibana.yml configuration file:

    opendistro_security.multitenancy.enabled: true

    When you access Kibana, a popup window asks you to define a tenant. You can avoid it by modifying the default route to include the selection of a tenant, for example, global.

    server.defaultRoute: /app/wazuh?security_tenant=global

  12. Restart Kibana:

    # systemctl daemon-reload
# systemctl enable kibana
# systemctl start kibana

  13. (For upgrades from 3.x versions) Once Kibana is accessible, remove the wazuh-alerts-3.x-* index pattern. Since Wazuh 4.0 it has been replaced by wazuh-alerts-* , it is necessary to remove the old pattern in order for the new one to take its place.

    # curl 'https://<kibana_ip>:<kibana_port>/api/saved_objects/index-pattern/wazuh-alerts-3.x-*' -X DELETE  -H 'Content-Type: application/json' -H 'kbn-version: 7.10.2' -k -uadmin:admin

    If you have a custom index pattern, be sure to replace it accordingly.

  14. Clear the browser's cache and cookies.

Finishing the upgrade

  1. Recommended action - Disable the Wazuh repository when finished upgrading the Wazuh installation in the node to prevent an upgrade to the newest Elastic Stack version due to the possibility of undoing changes with the Wazuh Kibana plugin.

    # sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/wazuh.repo

Next steps

The next step consists in upgrading the Wazuh agents.