Upgrading Elastic Stack from 6.8 to 7.x

Coming new in version Elastic 7.x, there is an architecture change introduced in Wazuh Stack. Logstash is no longer required, Filebeat will send the events directly to Elasticsearch server. In addition, Elasticsearch 7 has Java embedded, so unless you decide to use Logstash, Java is no longer required.

In case of upgrading from a version of Elasticsearch prior to 6.8, visit the Upgrading Elastic Stack from a legacy version section.

Prepare the Elastic Stack

  1. Stop the services:

    # systemctl stop filebeat
    # systemctl stop kibana
    
  2. Add the new repository for Elastic Stack 7.x:

    • CentOS/RHEL/Fedora:

      # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
      # cat > /etc/yum.repos.d/elastic.repo << EOF
      [elasticsearch-7.x]
      name=Elasticsearch repository for 7.x packages
      baseurl=https://artifacts.elastic.co/packages/7.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
      EOF
      
    • Debian/Ubuntu:

      # curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
      # echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list
      
    • openSUSE:

      # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
      # cat > /etc/zypp/repos.d/elastic.repo << EOF
      [elasticsearch-7.x]
      name=Elasticsearch repository for 7.x packages
      baseurl=https://artifacts.elastic.co/packages/7.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
      EOF
      

Upgrade Elasticsearch

In case of having previously upgraded from a legacy version (versions prior to 6.8), the steps from 1 to 3 must be omitted.

  1. Disable shard allocation.

    curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
    {
      "persistent": {
        "cluster.routing.allocation.enable": "primaries"
      }
    }
    '
    
  2. Stop non-essential indexing and perform a synced flush. (Optional)

    curl -X POST "localhost:9200/_flush/synced"
    
  3. Shut down a single node.

    # systemctl stop elasticsearch
    
  4. Upgrade the node you shut down.

    • CentOS/RHEL/Fedora:

      # yum install elasticsearch-7.9.2
      
    • Debian/Ubuntu:

      # apt-get install elasticsearch=7.9.2
      
  5. Starting with Elasticsearch 7.0, master nodes require a configuration setting with the list of the cluster master nodes. The following settings must be added to the configuration of the Elasticsearch master node (elasticsearch.yml).

    discovery.seed_hosts:
      - master_node_name_or_ip_address
    cluster.initial_master_nodes:
      - master_node_name_or_ip_address
    
  6. Restart the service.

    # systemctl daemon-reload
    # systemctl restart elasticsearch
    
  7. Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

    curl -X GET "localhost:9200/_cat/nodes"
    
  8. Reenable shard allocation.

    curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
    {
      "persistent": {
        "cluster.routing.allocation.enable": null
      }
    }
    '
    
  9. Before upgrading the next node, wait for the cluster to finish shard allocation.

    curl -X GET "localhost:9200/_cat/health?v"
    
  10. Repeat it for every Elasticsearch node.

Field migration: From @timestamp to timestamp

In previous Elastic search versions, the Elastic documents were indexed using the field @timestamp as the reference field for time-based indices. Starting in Elastic 7.x, this field has become a reserved field and it is no longer manipulable. Wazuh time-based indices now make use of field timestamp instead.

Due to this change, previous alerts won't be visible in Wazuh indices, an update must be performed to all previous indices in order to complete the upgrade.

Run below request for each Wazuh index created before Elastic 7.x upgrade. It will add the timestamp field for all the index documents.

Here is an example of how to run the request using the index wazuh-alerts-3.x-2019.05.16.

curl -X POST "localhost:9200/wazuh-alerts-3.x-2019.05.16/wazuh/_update_by_query?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "timestamp"
        }
      }
    }
  },
  "script": "ctx._source.timestamp = ctx._source[\"@timestamp\"]"
}
'

The request must be run for all previous indices you want to migrate, modify the date parameter according to your index name.

Upgrade Filebeat

  1. Upgrade Filebeat.

    • CentOS/RHEL/Fedora:

      # yum install filebeat-7.9.2
      
    • Debian/Ubuntu:

      # apt-get install filebeat=7.9.2
      
  2. Update the configuration file.

    # cp /etc/filebeat/filebeat.yml /backup/filebeat.yml.backup
    # curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v3.13.6/extensions/filebeat/7.x/filebeat.yml
    # chmod go+r /etc/filebeat/filebeat.yml
    
  3. Download the alerts template for Elasticsearch:

    # curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v3.13.6/extensions/elasticsearch/7.x/wazuh-template.json
    # chmod go+r /etc/filebeat/wazuh-template.json
    
  4. Download the Wazuh module for Filebeat:

    # curl -s https://packages.wazuh.com/3.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module
    
  5. Edit the file /etc/filebeat/filebeat.yml and replace YOUR_ELASTIC_SERVER_IP with the IP address or the hostname of the Elasticsearch server. For example:

    output.elasticsearch.hosts: ['http://YOUR_ELASTIC_SERVER_IP:9200']
    
  6. Restart Filebeat.

    # systemctl daemon-reload
    # systemctl restart filebeat
    

Upgrade Kibana

  1. Modify Kibana configuration file /etc/kibana/kibana.yml and replace elasticsearch.url: "address:9200" by elasticsearch.hosts: ["address:9200"].

  2. Remove the Wazuh app.

    # cd /usr/share/kibana/
    # sudo -u kibana bin/kibana-plugin remove wazuh
    
  3. Upgrade Kibana.

    • For CentOS/RHEL/Fedora:

      # yum install kibana-7.9.2
      
    • For Debian/Ubuntu:

      # apt-get install kibana=7.9.2
      
  4. Install the Wazuh app.

    • From URL:

      # cd /usr/share/kibana/
      # sudo -u kibana bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.13.6_7.9.2.zip
      
    • From the package:

      # cd /usr/share/kibana/
      # sudo -u kibana bin/kibana-plugin install file:///path/wazuhapp-3.13.6_7.9.2.zip
      
  5. For installations on Kibana 7.6.X versions it is recommended to increase the heap size of Kibana to ensure the Kibana's plugins installation:

    # cat >> /etc/default/kibana << EOF
    NODE_OPTIONS="--max_old_space_size=2048"
    EOF
    
  6. Restart Kibana.

    # systemctl daemon-reload
    # systemctl restart kibana
    

Disabling repositories

  • For CentOS/RHEL/Fedora:

    # sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo
    
  • For Debian/Ubuntu:

    # sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/elastic-7.x.list
    # apt-get update
    

    Alternatively, you can set the package state to hold, which will stop updates (although you can still upgrade it manually using apt-get install).

    # echo "elasticsearch hold" | sudo dpkg --set-selections
    # echo "kibana hold" | sudo dpkg --set-selections
    
  • For openSUSE:

    # sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/elastic.repo