Upgrading Elastic Stack from 6.8 to 7.x

Coming new in version Elastic 7.x, there is an architecture change introduced in Wazuh Stack. Logstash is no longer required, Filebeat will send the events directly to Elasticsearch server. In addition, Elasticsearch 7 has Java embedded, so unless you decide to use Logstash, Java is not longer required.

Prepare the Elastic Stack

  1. Stop the services:

    # systemctl stop filebeat
# systemctl stop kibana

  2. Add the new repository for Elastic Stack 7.x:

    • CentOS/RHEL/Fedora:

      # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
# cat > /etc/yum.repos.d/elastic.repo << EOF
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

    • Debian/Ubuntu:

      # curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
# echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list

Upgrade Elasticsearch

  1. Disable shard allocation

    curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
'

  2. Stop non-essential indexing and perform a synced flush. (Optional)

    curl -X POST "localhost:9200/_flush/synced"

  3. Shut down a single node.

    # systemctl stop elasticsearch

  4. Upgrade the node you shut down.

    • CentOS/RHEL/Fedora:

      # yum install elasticsearch-7.5.1

    • Debian/Ubuntu:

      # apt-get install elasticsearch=7.5.1
# systemctl restart elasticsearch

  5. Starting with Elasticsearch 7.0, master nodes require a configuration setting with the list of the cluster master nodes. The following settings must be added in the configuration of the Elasticsearch master node (elasticsearch.yml).

    discovery.seed_hosts:
  - master_node_name_or_ip_address
cluster.initial_master_nodes:
  - master_node_name_or_ip_address

  6. Restart the service.

    # systemctl daemon-reload
# systemctl restart elasticsearch

  7. Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

    curl -X GET "localhost:9200/_cat/nodes"

  8. Reenable shard allocation.

    curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}
'

  9. Before upgrading the next node, wait for the cluster to finish shard allocation.

    curl -X GET "localhost:9200/_cat/health?v"

  10. Repeat it for every Elasticsearch node.

Field migration: From @timestamp to timestamp

In previous Elastic search versions, the Elastic documents were indexed using the field @timestamp as the reference field for time-based indices. Starting in Elastic 7.x, this field has become a reserved field and it is no longer manipulable. Wazuh time-based indices now make use of field timestamp instead.

Due to this change, previous alerts won't be visible in Wazuh indices, an update must be performed to all previous indices in order to complete the upgrade.

Run below request for each Wazuh index created before Elastic 7.x upgrade. It will add the timestamp field for all the index documents.

Here is an example of how to run the request using the index wazuh-alerts-3.x-2019.05.16.

curl -X POST "localhost:9200/wazuh-alerts-3.x-2019.05.16/wazuh/_update_by_query?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "timestamp"
        }
      }
    }
  },
  "script": "ctx._source.timestamp = ctx._source[\"@timestamp\"]"
}
'

The request must be run for all previous indices you want to migrate, modify the date parameter according to your index name.

Upgrade Filebeat

  1. Upgrade Filebeat.

    • CentOS/RHEL/Fedora:

      # yum install filebeat-7.5.1

    • Debian/Ubuntu:

      # apt-get install filebeat=7.5.1

  2. Update the configuration file.

    # cp /etc/filebeat/filebeat.yml /backup/filebeat.yml.backup
# curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v3.10.2/extensions/filebeat/7.x/filebeat.yml
# chmod go+r /etc/filebeat/filebeat.yml

  3. Download the Wazuh module for Filebeat:

    # curl -s https://packages.wazuh.com/3.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module

  4. Edit the file /etc/filebeat/filebeat.yml and replace YOUR_ELASTIC_SERVER_IP with the IP address or the hostname of the Elasticsearch server. For example:

    output.elasticsearch.hosts: ['http://YOUR_ELASTIC_SERVER_IP:9200']

  5. Restart Filebeat.

    # systemctl daemon-reload
# systemctl restart filebeat

Upgrade Kibana

  1. Modify Kibana configuration file /etc/kibana/kibana.yml and replace elasticsearch.url: "address:9200" by elasticsearch.hosts: ["address:9200"].

  2. Remove the Wazuh app.

    # /usr/share/kibana/bin/kibana-plugin remove wazuh

  3. Upgrade Kibana.

    • For CentOS/RHEL/Fedora:

      # yum install kibana-7.5.1

    • For Debian/Ubuntu:

      # apt-get install kibana=7.5.1

  4. Install the Wazuh app.

    # sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.10.2_7.5.1.zip

  5. Restart Kibana.

    # systemctl daemon-reload
# systemctl restart kibana