Upgrading Elastic Stack from 6.8 to 7.x

Preparing the Elastic Stack

1. Stop the services:

   # systemctl stop filebeat
   # systemctl stop kibana
   

2. Add the new repository for Elastic Stack 7.x:

   YUMAPTZYpp

   # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
   # cat > /etc/yum.repos.d/elastic.repo << EOF
   [elasticsearch-7.x]
   name=Elasticsearch repository for 7.x packages
   baseurl=https://artifacts.elastic.co/packages/7.x/yum
   gpgcheck=1
   gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
   enabled=1
   autorefresh=1
   type=rpm-md
   EOF
   

Upgrading Elasticsearch

1. Disable shard allocation:

   curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
   {
     "persistent": {
       "cluster.routing.allocation.enable": "primaries"
     }
   }
   '
   

2. Stop non-essential indexing and perform a synced flush (optional):

   curl -X POST "localhost:9200/_flush/synced"
   

3. Shut down a single node:

   # systemctl stop elasticsearch
   

4. Upgrade the shut down node:

   YUMAPTzypper

   # yum install elasticsearch-7.9.1
   

5. Starting with Elasticsearch 7.0, master nodes require a configuration setting with the list of the cluster master nodes. The following settings must be added in the configuration of the Elasticsearch master node (elasticsearch.yml):

   discovery.seed_hosts:
     - master_node_name_or_ip_address
   cluster.initial_master_nodes:
     - master_node_name_or_ip_address
   

6. Restart the service:

   # systemctl daemon-reload
   # systemctl restart elasticsearch
   

7. Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

   curl -X GET "localhost:9200/_cat/nodes"
   

8. Reenable shard allocation:

   curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
   {
     "persistent": {
       "cluster.routing.allocation.enable": null
     }
   }
   '
   

9. Before upgrading the next node, wait for the cluster to finish shard allocation:

   curl -X GET "localhost:9200/_cat/health?v"
   

10. Repeat the steps for every Elasticsearch node.

Field migration: From @timestamp to timestamp

In the previous Elasticsearch versions, the Elastic documents were indexed using the @timestamp field as the reference field for time-based indices. Starting in Elastic 7.x, this field has become a reserved field and it is no longer manipulable. The Wazuh time-based indices use the timestamp field instead.

Due to this change, the previous alerts will not be visible in the Wazuh indices and update must be performed to all previous indices in order to complete the upgrade.

Run the request below for each Wazuh index created before the Elastic 7.x upgrade. It will add the timestamp field for all the index documents.

An example of how to run the request using the index wazuh-alerts-3.x-2019.05.16 looks as follows:

curl -X POST "localhost:9200/wazuh-alerts-3.x-2019.05.16/wazuh/_update_by_query?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "timestamp"
        }
      }
    }
  },
  "script": "ctx._source.timestamp = ctx._source[\"@timestamp\"]"
}
'

The request must be executed for all previous indices which need to be migrated. Modify the date parameter according to the index name.

More information about the request can be found in the Update by query section of the Elasticsearch documentation.

Upgrading Filebeat

1. Upgrade Filebeat:

   YUMAPTZYpp

   # yum install filebeat-7.9.1
   

2. Update the Filebeat configuration file:

   # cp /etc/filebeat/filebeat.yml <back-up-directory>/filebeat.yml.backup
   # curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/filebeat/7.x/filebeat.yml
   # chmod go+r /etc/filebeat/filebeat.yml
   

3. Download the alerts template for Elasticsearch:

   # curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/elasticsearch/7.x/wazuh-template.json
   # chmod go+r /etc/filebeat/wazuh-template.json
   

4. Download the Wazuh module for Filebeat:

   # curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module
   

5. Edit the /etc/filebeat/filebeat.yml configuration file and replace YOUR_ELASTIC_SERVER_IP with the IP address or the hostname of the Elasticsearch server:

   output.elasticsearch.hosts: ['http://YOUR_ELASTIC_SERVER_IP:9200']
   

6. Restart Filebeat:

   # systemctl daemon-reload
   # systemctl restart filebeat
   

Upgrading Kibana

1. Modify the /etc/kibana/kibana.yml configuration file and replace elasticsearch.url: "address:9200" by elasticsearch.hosts: ["address:9200"].

2. Remove the Wazuh Kibana plugin:

   # cd /usr/share/kibana/
   # sudo -u kibana bin/kibana-plugin remove wazuh
   

3. Upgrade Kibana:

   YUMAPTZYpp

   # yum install kibana-7.9.1
   

4. Install the Wazuh Kibana plugin:

   From URLFrom the package

   # cd /usr/share/kibana/
   # sudo -u kibana bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-4.0.4_7.9.1.zip
   

5. For installations on Kibana 7.6.x version and higher, it is recommended to increase the heap size of Kibana to ensure the Kibana's plugins installation:

   # cat >> /etc/default/kibana << EOF
   NODE_OPTIONS="--max_old_space_size=2048"
   EOF
   

6. Restart Kibana:

   # systemctl daemon-reload
   # systemctl restart kibana