Upgrading Elastic Stack from 6.8 to 7.x
This section guides through the upgrade process of Elastic Stack components including Elasticsearch, Filebeat and Kibana for the Elastic distribution.
Coming new in Elastic 7.x, there is an architecture change introduced in the Wazuh installation. Logstash is no longer required and Filebeat will send the events directly to the Elasticsearch. In addition, Elasticsearch 7.x has Java embedded, so unless the user decides to use Logstash, Java is no longer required.
Preparing the Elastic Stack
Stop the services:
# systemctl stop filebeat # systemctl stop kibana
Add the new repository for Elastic Stack 7.x:
# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch # cat > /etc/yum.repos.d/elastic.repo << EOF [elasticsearch-7.x] name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF
# curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add - # echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list
# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch # cat > /etc/zypp/repos.d/elastic.repo << EOF [elasticsearch-7.x] name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF
Upgrading Elasticsearch
Disable shard allocation:
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.enable": "primaries" } } '
Stop non-essential indexing and perform a synced flush (optional):
curl -X POST "localhost:9200/_flush/synced"
Shut down a single node:
# systemctl stop elasticsearch
Upgrade the shut down node:
# yum install elasticsearch-7.9.1
# apt-get install elasticsearch=7.9.1 # systemctl restart elasticsearch
# zypper update elasticsearch-7.9.1
Starting with Elasticsearch 7.0, master nodes require a configuration setting with the list of the cluster master nodes. The following settings must be added in the configuration of the Elasticsearch master node (
elasticsearch.yml
):discovery.seed_hosts: - master_node_name_or_ip_address cluster.initial_master_nodes: - master_node_name_or_ip_address
Restart the service:
# systemctl daemon-reload # systemctl restart elasticsearch
Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a
_cat/nodes
request:curl -X GET "localhost:9200/_cat/nodes"
Reenable shard allocation:
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.enable": null } } '
Before upgrading the next node, wait for the cluster to finish shard allocation:
curl -X GET "localhost:9200/_cat/health?v"
Repeat the steps for every Elasticsearch node.
Field migration: From @timestamp to timestamp
In the previous Elasticsearch versions, the Elastic documents were indexed using the @timestamp
field as the reference field for time-based indices. Starting in Elastic 7.x, this field has become a reserved field and it is no longer manipulable. The Wazuh time-based indices use the timestamp
field instead.
Due to this change, the previous alerts will not be visible in the Wazuh indices and update must be performed to all previous indices in order to complete the upgrade.
Run the request below for each Wazuh index created before the Elastic 7.x upgrade. It will add the timestamp
field for all the index documents.
An example of how to run the request using the index wazuh-alerts-3.x-2019.05.16
looks as follows:
curl -X POST "localhost:9200/wazuh-alerts-3.x-2019.05.16/wazuh/_update_by_query?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "timestamp"
}
}
}
},
"script": "ctx._source.timestamp = ctx._source[\"@timestamp\"]"
}
'
The request must be executed for all previous indices which need to be migrated. Modify the date parameter according to the index name.
More information about the request can be found in the Update by query section of the Elasticsearch documentation.
Upgrading Filebeat
Upgrade Filebeat:
# yum install filebeat-7.9.1
# apt-get install filebeat=7.9.1
# zypper update filebeat-7.9.1
Update the Filebeat configuration file:
# cp /etc/filebeat/filebeat.yml <back-up-directory>/filebeat.yml.backup # curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/filebeat/7.x/filebeat.yml # chmod go+r /etc/filebeat/filebeat.yml
Download the alerts template for Elasticsearch:
# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/elasticsearch/7.x/wazuh-template.json # chmod go+r /etc/filebeat/wazuh-template.json
Download the Wazuh module for Filebeat:
# curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module
Edit the
/etc/filebeat/filebeat.yml
configuration file and replaceYOUR_ELASTIC_SERVER_IP
with the IP address or the hostname of the Elasticsearch server:output.elasticsearch.hosts: ['http://YOUR_ELASTIC_SERVER_IP:9200']
Restart Filebeat:
# systemctl daemon-reload # systemctl restart filebeat
Upgrading Kibana
Modify the
/etc/kibana/kibana.yml
configuration file and replaceelasticsearch.url: "address:9200"
byelasticsearch.hosts: ["address:9200"]
.Remove the Wazuh Kibana plugin:
# cd /usr/share/kibana/ # sudo -u kibana bin/kibana-plugin remove wazuh
Upgrade Kibana:
# yum install kibana-7.9.1
# apt-get install kibana=7.9.1
# zypper update kibana-7.9.1
Install the Wazuh Kibana plugin:
# cd /usr/share/kibana/ # sudo -u kibana bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-4.0.4_7.9.1.zip
# cd /usr/share/kibana/ # sudo -u kibana bin/kibana-plugin install file:///path/wazuhapp-4.0.4_7.9.1.zip
For installations on Kibana 7.6.x version and higher, it is recommended to increase the heap size of Kibana to ensure the Kibana's plugins installation:
# cat >> /etc/default/kibana << EOF NODE_OPTIONS="--max_old_space_size=2048" EOF
Restart Kibana:
# systemctl daemon-reload # systemctl restart kibana
Disabling the repositories
It is recommended to disable the Elastic repository to prevent an upgrade to a newer Elastic Stack version due to the possibility of undoing changes with the Wazuh Kibana plugin:
# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/elastic-7.x.list # apt-get updateAlternatively, the user can set the package state to
hold
, which will stop updates. It will be still possible to upgrade it manually usingapt-get install
:# echo "elasticsearch hold" | sudo dpkg --set-selections # echo "filebeat hold" | sudo dpkg --set-selections # echo "kibana hold" | sudo dpkg --set-selections# sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/elastic.repo