Upgrading Open Distro for Elasticsearch

This section guides through the upgrade process of Elasticsearch, Filebeat and Kibana for Open Distro for Elasticsearch distribution.

Note

Root user privileges are required to execute all the commands described below.

Preparing Open Distro for Elasticsearch

  1. Stop the services:

    # systemctl stop filebeat
    # systemctl stop kibana
    
    # service filebeat stop
    # service kibana stop
    
  2. Prepare the repositories. Wazuh now hosts the Open Distro packages. In order to prevent accidental upgrades, it is recommended to disable the Open Distro repository. Besides, if Elastic repository was added, it is also recommended to disable it:

2.1. Disable the Open Distro for Elasticsearch repository:

# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/opendistroforelasticsearch-artifacts.repo
# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/opendistroforelasticsearch.list
# apt-get update
# sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/opendistroforelasticsearch-artifacts.repo

2.2. (Optional) Disable the Elastic Stack basic license repository:

# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo
# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/elastic-7.x.list
# apt-get update

Alternatively, the user can set the package state to hold, which will stop updates. It will be still possible to upgrade it manually using apt-get install:

# echo "elasticsearch hold" | sudo dpkg --set-selections
# echo "filebeat hold" | sudo dpkg --set-selections
# echo "kibana hold" | sudo dpkg --set-selections
# sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/elastic.repo

2.3. Add the Wazuh 4.x repository:

  1. Import the GPG key:

    # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
    
  2. Add the repository:

    # cat > /etc/yum.repos.d/wazuh.repo << EOF
    [wazuh]
    gpgcheck=1
    gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
    enabled=1
    name=EL-\$releasever - Wazuh
    baseurl=https://packages.wazuh.com/4.x/yum/
    protect=1
    EOF
    
  1. Clean the YUM cache:

# yum clean all
  1. Install the GPG key:

    # curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
    
  2. Add the repository:

    # echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
    
  3. Update the package information:

    # apt-get update
    
  1. Import the GPG key:

    # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
    
  2. Add the repository:

    # cat > /etc/zypp/repos.d/wazuh.repo <<\EOF
    [wazuh]
    gpgcheck=1
    gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
    enabled=1
    name=EL-$releasever - Wazuh
    baseurl=https://packages.wazuh.com/4.x/yum/
    protect=1
    EOF
    

Upgrading Elasticsearch

This guide explains how to perform a rolling upgrade, which allows you to shut down one node at a time for minimal disruption of service. The cluster remains available throughout the process.

In the commands below 127.0.0.1 IP address is used. If Elasticsearch is bound to a specific IP address, replace 127.0.0.1 with your Elasticsearch IP. If using http, the option -k must be omitted and if not using user/password authentication, -u must be omitted.

  1. Disable shard allocation:

    curl -X PUT "https://127.0.0.1:9200/_cluster/settings"  -u <username>:<password> -k -H 'Content-Type: application/json' -d'
    {
      "persistent": {
        "cluster.routing.allocation.enable": "primaries"
      }
    }
    '
    
  2. Stop non-essential indexing and perform a synced flush:

    curl -X POST "https://127.0.0.1:9200/_flush/synced" -u <username>:<password> -k
    
  3. Shut down a single node:

    # systemctl stop elasticsearch
    
    # service elasticsearch stop
    
  4. Upgrade the node you shut down:

    # yum install opendistroforelasticsearch-1.13.2
    

    Upgrade Elasticsearch OSS:

    # apt install elasticsearch-oss=7.10.2
    

    Upgrade Open Distro for Elasticsearch:

    # apt install opendistroforelasticsearch=1.13.2-1
    
    # zypper update opendistroforelasticsearch-1.13.2
    
  5. Restart the service:

    # systemctl daemon-reload
    # systemctl enable elasticsearch
    # systemctl start elasticsearch
    

    Choose one option according to the OS used:

    1. Debian based OS

    # update-rc.d elasticsearch defaults 95 10
    # service elasticsearch start
    
    1. RPM based OS

    # chkconfig --add elasticsearch
    # service elasticsearch start
    
  6. Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

    curl -X GET "https://127.0.0.1:9200/_cat/nodes" -u <username>:<password> -k
    
  7. Reenable shard allocation:

    curl -X PUT "https://127.0.0.1:9200/_cluster/settings" -u <username>:<password> -k -H 'Content-Type: application/json' -d'
    {
      "persistent": {
        "cluster.routing.allocation.enable": "all"
      }
    }
    '
    
  8. Before upgrading the next node, wait for the cluster to finish shard allocation:

    curl -X GET "https://127.0.0.1:9200/_cat/health?v" -u <username>:<password> -k
    
  9. Repeat the steps for every Elasticsearch node.

Note

The Open Distro for Elasticsearch performance analyzer plugin is installed by default and can have a negative impact on system resources. We recommend removing it with the following command /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro-performance-analyzer.

Upgrading Filebeat

  1. Upgrade Filebeat:

    # yum install filebeat-7.10.2
    
    # apt-get install filebeat=7.10.2
    
    # zypper update filebeat-7.10.2
    
  2. Download the alerts template for Elasticsearch:

    # curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.2.4/extensions/elasticsearch/7.x/wazuh-template.json
    # chmod go+r /etc/filebeat/wazuh-template.json
    
  3. Download the Wazuh module for Filebeat:

    # curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module
    
  4. Edit the /etc/filebeat/filebeat.yml configuration file. This step is only needed for the upgrade of a Distributed installation. In case of having an All-in-one installation, the file is already configured:

    output.elasticsearch:
      hosts: ["<elasticsearch_ip>:9200"]
    

    Replace elasticsearch_ip with the IP address or the hostname of the Elasticsearch server.

    output.elasticsearch:
      hosts: ["<elasticsearch_ip_node_1>:9200", "<elasticsearch_ip_node_2>:9200", "<elasticsearch_ip_node_3>:9200"]
    

    Replace elasticsearch_ip_node_x with the IP address or the hostname of the Elasticsearch server to connect to.

    During the installation, the default username and password were used. If those credentials were changed, replace those values in the filebeat.yml configuration file.

  5. Restart Filebeat:

    # systemctl daemon-reload
    # systemctl enable filebeat
    # systemctl start filebeat
    

    Choose one option according to the OS used:

    1. Debian based OS

    # update-rc.d filebeat defaults 95 10
    # service filebeat start
    
    1. RPM based OS

    # chkconfig --add filebeat
    # service filebeat start
    
  6. Upload the new Wazuh template to Elasticsearch. This step can be omitted in Wazuh single-node installations:

# filebeat setup --index-management -E output.logstash.enabled=false

Upgrading Kibana

Warning

The location of the Wazuh Kibana plugin configuration file has been moved to /usr/share/kibana/data/wazuh/config/wazuh.yml

  1. Copy the Wazuh Kibana plugin configuration file to its new location:

    Create the new directory and copy the Wazuh Kibana plugin configuration file.

    # mkdir -p /usr/share/kibana/data/wazuh/config/
    # cp /usr/share/kibana/optimize/wazuh/config/wazuh.yml /usr/share/kibana/data/wazuh/config/wazuh.yml
    

    Create the new directory and copy the Wazuh Kibana plugin configuration file:

    # mkdir -p /usr/share/kibana/data/wazuh/config/
    # cp /usr/share/kibana/plugins/wazuh/wazuh.yml /usr/share/kibana/data/wazuh/config/wazuh.yml
    

    Create the new directory and copy the Wazuh Kibana plugin configuration file:

    # mkdir -p /usr/share/kibana/data/wazuh/config/
    # cp /usr/share/kibana/plugins/wazuh/config.yml /usr/share/kibana/data/wazuh/config/wazuh.yml
    

    Edit the /usr/share/kibana/data/wazuh/config/wazuh.yml configuration file and add to the end of the file the following default structure to define an Wazuh API entry:

    hosts:
      - <id>:
         url: http(s)://<api_url>
         port: <api_port>
         username: <api_user>
         password: <api_password>
         run_as: false
    

    The following values need to be replaced:

    • <id>: an arbitrary ID.

    • <api_url>: url of the Wazuh API.

    • <api_port>: port.

    • <api_user>: credentials to authenticate.

    • <api_password>: credentials to authenticate.

    In case of having more Wazuh API entries, each of them must be added manually.

  2. (For upgrades from 3.x versions) Replace the value user by username and set the username and password as wazuh-wui in the file /usr/share/kibana/data/wazuh/config/wazuh.yml:

    hosts:
      - default:
          url: https://localhost
          port: 55000
          username: wazuh-wui
          password: wazuh-wui
          run_as: false
    
  3. Remove the Wazuh Kibana plugin:

    # cd /usr/share/kibana/
    # sudo -u kibana bin/kibana-plugin remove wazuh
    
  4. Upgrade Kibana:

    # yum install opendistroforelasticsearch-kibana-1.13.2
    
    # apt-get install opendistroforelasticsearch-kibana=1.13.2
    
    # zypper update opendistroforelasticsearch-kibana-1.13.2
    
  5. (For upgrades from 3.x versions) Remove generated bundles and the wazuh-registry.json file:

    # rm -rf /usr/share/kibana/optimize/bundles
    # rm -f /usr/share/kibana/optimize/wazuh/config/wazuh-registry.json
    
  6. Update file permissions. This will prevent errors when generating new bundles or updating the Wazuh Kibana plugin:

    # chown -R kibana:kibana /usr/share/kibana/data
    # chown -R kibana:kibana /usr/share/kibana/plugins
    
  7. Install the Wazuh Kibana plugin:

    # cd /usr/share/kibana/
    # sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/4.x/ui/kibana/wazuh_kibana-4.2.4_7.10.2-1.zip
    
    # cd /usr/share/kibana/
    # sudo -u kibana bin/kibana-plugin install file:///path/wazuh_kibana-4.2.4_7.10.2-1.zip
    
  8. Update configuration file permissions:

    # sudo chown kibana:kibana /usr/share/kibana/data/wazuh/config/wazuh.yml
    # sudo chmod 600 /usr/share/kibana/data/wazuh/config/wazuh.yml
    
  9. For installations on Kibana 7.6.x version and higher, it is recommended to increase the heap size of Kibana to ensure the Kibana’s plugins installation:

    # cat >> /etc/default/kibana << EOF
    NODE_OPTIONS="--max_old_space_size=2048"
    EOF
    
  10. Link Kibana’s socket to privileged port 443:

    # setcap 'cap_net_bind_service=+ep' /usr/share/kibana/node/bin/node
    
  11. (Recommended) Enable multitenancy in the /etc/kibana/kibana.yml configuration file:

    opendistro_security.multitenancy.enabled: true
    

    When you access Kibana, a popup window asks you to define a tenant. You can avoid it by modifying the default route to include the selection of a tenant, for example, global.

    server.defaultRoute: /app/wazuh?security_tenant=global
    
  12. Restart Kibana:

    # systemctl daemon-reload
    # systemctl enable kibana
    # systemctl start kibana
    

    Choose one option according to the OS used:

    1. Debian based OS

    # update-rc.d kibana defaults 95 10
    # service kibana start
    
    1. RPM based OS

    # chkconfig --add kibana
    # service kibana start
    
  13. (For upgrades from 3.x versions) Once Kibana is accesible, remove the wazuh-alerts-3.x-* index pattern. Since Wazuh 4.0 it has been replaced by wazuh-alerts-* , it is necessary to remove the old pattern in order for the new one to take its place.

    # curl 'https://<kibana_ip>:<kibana_port>/api/saved_objects/index-pattern/wazuh-alerts-3.x-*' -X DELETE  -H 'Content-Type: application/json' -H 'kbn-version: 7.10.2' -k -uadmin:admin
    

    If you have a custom index pattern, be sure to replace it accordingly.

  14. Clear the browser’s cache and cookies.

Disabling the repository

It is recommended to disable the Wazuh repository to prevent an upgrade to a newest Elastic Stack version due to the possibility of undoing changes with the Wazuh Kibana plugin:

# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/wazuh.repo
# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/wazuh.list
# apt-get update
# sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/wazuh.repo

Next step

The next step consists on upgrading the Wazuh agents.