Upgrading Elastic Stack from 6.8 to 7.x

This section guides through the upgrade process of Elastic Stack components including Elasticsearch, Filebeat and Kibana for the Elastic distribution.

Coming new in Elastic 7.x, there is an architecture change introduced in the Wazuh installation. Logstash is no longer required and Filebeat will send the events directly to the Elasticsearch. In addition, Elasticsearch 7.x has Java embedded, so unless the user decides to use Logstash, Java is no longer required.

Preparing the Elastic Stack

  1. Stop the services:

    # systemctl stop filebeat
    # systemctl stop kibana
    
  2. Add the new repository for Elastic Stack 7.x:

    # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    # cat > /etc/yum.repos.d/elastic.repo << EOF
    [elasticsearch-7.x]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    EOF
    
    # curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
    # echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list
    
    # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    # cat > /etc/zypp/repos.d/elastic.repo << EOF
    [elasticsearch-7.x]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    EOF
    

Upgrading Elasticsearch

  1. Disable shard allocation:

    curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
    {
      "persistent": {
        "cluster.routing.allocation.enable": "primaries"
      }
    }
    '
    
  2. Stop non-essential indexing and perform a synced flush (optional):

    curl -X POST "localhost:9200/_flush/synced"
    
  3. Shut down a single node:

    # systemctl stop elasticsearch
    
  4. Upgrade the shut down node:

    # yum install elasticsearch-7.9.1
    
    # apt-get install elasticsearch=7.9.1
    # systemctl restart elasticsearch
    
    # zypper update elasticsearch-7.9.1
    
  5. Starting with Elasticsearch 7.0, master nodes require a configuration setting with the list of the cluster master nodes. The following settings must be added in the configuration of the Elasticsearch master node (elasticsearch.yml):

    discovery.seed_hosts:
      - master_node_name_or_ip_address
    cluster.initial_master_nodes:
      - master_node_name_or_ip_address
    
  6. Restart the service:

    # systemctl daemon-reload
    # systemctl restart elasticsearch
    
  7. Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

    curl -X GET "localhost:9200/_cat/nodes"
    
  8. Reenable shard allocation:

    curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
    {
      "persistent": {
        "cluster.routing.allocation.enable": null
      }
    }
    '
    
  9. Before upgrading the next node, wait for the cluster to finish shard allocation:

    curl -X GET "localhost:9200/_cat/health?v"
    
  10. Repeat the steps for every Elasticsearch node.

Field migration: From @timestamp to timestamp

In the previous Elasticsearch versions, the Elastic documents were indexed using the @timestamp field as the reference field for time-based indices. Starting in Elastic 7.x, this field has become a reserved field and it is no longer manipulable. The Wazuh time-based indices use the timestamp field instead.

Due to this change, the previous alerts will not be visible in the Wazuh indices and update must be performed to all previous indices in order to complete the upgrade.

Run the request below for each Wazuh index created before the Elastic 7.x upgrade. It will add the timestamp field for all the index documents.

An example of how to run the request using the index wazuh-alerts-3.x-2019.05.16 looks as follows:

curl -X POST "localhost:9200/wazuh-alerts-3.x-2019.05.16/wazuh/_update_by_query?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "timestamp"
        }
      }
    }
  },
  "script": "ctx._source.timestamp = ctx._source[\"@timestamp\"]"
}
'

The request must be executed for all previous indices which need to be migrated. Modify the date parameter according to the index name.

More information about the request can be found in the Update by query section of the Elasticsearch documentation.

Upgrading Filebeat

  1. Upgrade Filebeat:

    # yum install filebeat-7.9.1
    
    # apt-get install filebeat=7.9.1
    
    # zypper update filebeat-7.9.1
    
  2. Update the Filebeat configuration file:

    # cp /etc/filebeat/filebeat.yml <back-up-directory>/filebeat.yml.backup
    # curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/filebeat/7.x/filebeat.yml
    # chmod go+r /etc/filebeat/filebeat.yml
    
  3. Download the alerts template for Elasticsearch:

    # curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/elasticsearch/7.x/wazuh-template.json
    # chmod go+r /etc/filebeat/wazuh-template.json
    
  4. Download the Wazuh module for Filebeat:

    # curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module
    
  5. Edit the /etc/filebeat/filebeat.yml configuration file and replace YOUR_ELASTIC_SERVER_IP with the IP address or the hostname of the Elasticsearch server:

    output.elasticsearch.hosts: ['http://YOUR_ELASTIC_SERVER_IP:9200']
    
  6. Restart Filebeat:

    # systemctl daemon-reload
    # systemctl restart filebeat
    

Upgrading Kibana

  1. Modify the /etc/kibana/kibana.yml configuration file and replace elasticsearch.url: "address:9200" by elasticsearch.hosts: ["address:9200"].

  2. Remove the Wazuh Kibana plugin:

    # cd /usr/share/kibana/
    # sudo -u kibana bin/kibana-plugin remove wazuh
    
  3. Upgrade Kibana:

    # yum install kibana-7.9.1
    
    # apt-get install kibana=7.9.1
    
    # zypper update kibana-7.9.1
    
  4. Install the Wazuh Kibana plugin:

    # cd /usr/share/kibana/
    # sudo -u kibana bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-4.0.4_7.9.1.zip
    
    # cd /usr/share/kibana/
    # sudo -u kibana bin/kibana-plugin install file:///path/wazuhapp-4.0.4_7.9.1.zip
    
  5. For installations on Kibana 7.6.x version and higher, it is recommended to increase the heap size of Kibana to ensure the Kibana’s plugins installation:

    # cat >> /etc/default/kibana << EOF
    NODE_OPTIONS="--max_old_space_size=2048"
    EOF
    
  6. Restart Kibana:

    # systemctl daemon-reload
    # systemctl restart kibana
    

Disabling the repositories

It is recommended to disable the Elastic repository to prevent an upgrade to a newer Elastic Stack version due to the possibility of undoing changes with the Wazuh Kibana plugin:

# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo
# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/elastic-7.x.list
# apt-get update

Alternatively, the user can set the package state to hold, which will stop updates. It will be still possible to upgrade it manually using apt-get install:

# echo "elasticsearch hold" | sudo dpkg --set-selections
# echo "filebeat hold" | sudo dpkg --set-selections
# echo "kibana hold" | sudo dpkg --set-selections
# sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/elastic.repo