Elasticsearch multi-node cluster
Open Distro for Elasticsearch is an open source distribution of Elasticsearch, a highly scalable full-text search engine. It offers advanced security, alerting, index management, deep performance analysis, and several other additional features.
This document will explain how to install Elasticsearch in a multi-node cluster. For resilience in case Elasticsearch nodes become unavailable, it is recommended to have an odd number of master eligible nodes, please take this into consideration when deciding the configuration of your Elasticsearch cluster.
Note
Root user privileges are necessary to run all the commands described below.
Installing Elasticsearch
The installation process for a multi-node cluster will be explained in three parts. The first one refers to the configuration of the initial node, in which the SSL certificates that will be deployed to the subsequent nodes are generated.
The second part will explain how to configure the remaining nodes of the cluster. Finally, the third part provides instructions for initializing the Elasticsearch cluster and verifying that everything is working properly.
Initial node
The following instructions are meant to be performed on the first Elasticsearch node to be configured.
Prerequisites
Open Distro for Elasticsearch requires the Java Development Kit and other packages installation including wget
, curl
, and unzip
that will be used in further steps:
Install all the necessary packages for the installation:
# export JAVA_HOME=/usr/ && yum install curl unzip wget && yum install java-11-openjdk-devel
In case JDK 11 is not available for the operating system being used, install the package adoptopenjdk-11-hotspot
using Adopt Open JDK.
Install all the necessary packages for the installation:
# apt install curl apt-transport-https unzip wget software-properties-common
Add the repository for Java Development Kit (JDK):
For Debian:
# echo 'deb http://deb.debian.org/debian stretch-backports main' > /etc/apt/sources.list.d/backports.listFor Ubuntu and other Debian based OS:
# add-apt-repository ppa:openjdk-r/ppa
Update repository data:
# apt update
Install all the required utilities:
# export JAVA_HOME=/usr/ && apt install openjdk-11-jdk
In case JDK 11 is not available for the operating system being used, install the package adoptopenjdk-11-hotspot
using Adopt Open JDK.
Install all the necessary packages for the installation:
# export JAVA_HOME=/usr/ && zypper install curl unzip wget && zypper install java-11-openjdk-devel
In case JDK 11 is not available for the operating system being used, install the package adoptopenjdk-11-hotspot
using Adopt Open JDK.
Adding the Wazuh repository
Import the GPG key:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
Add the repository:
# cat > /etc/yum.repos.d/wazuh.repo << EOF [wazuh] gpgcheck=1 gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH enabled=1 name=EL-$releasever - Wazuh baseurl=https://packages.wazuh.com/4.x/yum/ protect=1 EOF
Install the GPG key:
# curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
Add the repository:
# echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
Update the package information:
# apt-get update
Import the GPG key:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
Add the repository:
# cat > /etc/zypp/repos.d/wazuh.repo <<\EOF [wazuh] gpgcheck=1 gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH enabled=1 name=EL-$releasever - Wazuh baseurl=https://packages.wazuh.com/4.x/yum/ protect=1 EOF
Elasticsearch installation
Install Open Distro for Elasticsearch:
Install Open Distro for Elasticsearch:
# yum install opendistroforelasticsearch-1.11.0-1
Install Elasticsearch OSS and Open Distro for Elasticsearch:
# apt install elasticsearch-oss=7.9.1 opendistro-alerting=1.11.0.1-1 opendistro-anomaly-detection=1.11.0.0-1 opendistro-index-management=1.11.0.0-1 opendistro-job-scheduler=1.11.0.0-1 opendistro-knn=1.11.0.0-1 opendistro-knnlib=1.11.0.0 opendistro-performance-analyzer=1.11.0.0-1 opendistro-security=1.11.0.0-0 opendistro-sql=1.11.0.0-1 opendistroforelasticsearch=1.11.0-1
Install Open Distro for Elasticsearch:
# zypper install opendistroforelasticsearch-1.11.0-1
Elasticsearch configuration
To configure Elasticsearch download the following file /etc/elasticsearch/elasticsearch.yml
:
# curl -so /etc/elasticsearch/elasticsearch.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/elasticsearch/7.x/elasticsearch_cluster_initial_node.yml
Edit the file /etc/elasticsearch/elasticsearch.yml
. By default, the pre-configured template includes 3 nodes, in case of having a different number of nodes, lines must be added or removed accordingly:
network.host: <elasticsearch_ip>
node.name: node-1
cluster.name: <elastic_cluster>
cluster.initial_master_nodes:
- node-1
- node-2
- node-3
discovery.seed_hosts:
- <elasticsearch_ip_node1>
- <elasticsearch_ip_node2>
- <elasticsearch_ip_node3>
Depending on the node type, some parameters may vary between nodes. The cluster.initial_master_nodes
and the discovery.seed_hosts
are lists of all the master-eligible nodes in the cluster. The parameter node.master: false
must be included in every Elasticsearch node that will not be configured as master.
Values to be replaced:
<elasticsearch_ip>
: the host's IP. E.g.:10.0.0.2
.<elastic_cluster>
: Elasticsearch cluster name. E.g.:elastic-cluster-production
.<elasticsearch_ip_nodeX>
Elasticsearch cluster master-eligible nodes IP. E.g.:10.0.0.3
.The node certificates for each node must be specified under the
opendistro_security.nodes_dn
section.opendistro_security.nodes_dn: - CN=node-1,OU=Docu,O=Wazuh,L=California,C=US - CN=node-2,OU=Docu,O=Wazuh,L=California,C=US - CN=node-3,OU=Docu,O=Wazuh,L=California,C=US - CN=<common_name>,OU=<operational_unit>,O=<organization_name>,L=<locality>,C=<country_code>
Elasticsearch roles and users
In order to use the Wazuh Kibana plugin properly, it is necessary to add the extra roles and users:
# curl -so /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/elasticsearch/roles/roles.yml
# curl -so /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/elasticsearch/roles/roles_mapping.yml
# curl -so /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/elasticsearch/roles/internal_users.yml
The commands above add the following Wazuh users in Kibana:
wazuh_user |
Created for users who need read-only access to the Wazuh Kibana plugin. |
wazuh_admin |
Recommended user for users who need administrative privileges. |
Two additional roles are added, whose function is to give the appropriate permissions to users:
wazuh_ui_user |
This role provides |
wazuh_ui_admin |
This role allows |
These users and roles are designed to operate along with the Wazuh Kibana plugin and they are protected so they cannot be modified from the Kibana’s interface. To modify them or add new users or roles, the securityadmin
script has to be run.
Certificates creation and deployment
Remove the demo certificates:
# rm /etc/elasticsearch/esnode-key.pem /etc/elasticsearch/esnode.pem /etc/elasticsearch/kirk-key.pem /etc/elasticsearch/kirk.pem /etc/elasticsearch/root-ca.pem -f
Generate and deploy the certificates:
Move to the installation location and create the certificates directory:
# mkdir /etc/elasticsearch/certs # cd /etc/elasticsearch/certs
Download the Search Guard offline TLS tool to create the certificates:
# curl -so ~/search-guard-tlstool-1.8.zip https://maven.search-guard.com/search-guard-tlstool/1.8/search-guard-tlstool-1.8.zip
Extract the downloaded file. It is assumed that it has been downloaded in
~/
(home directory):# unzip ~/search-guard-tlstool-1.8.zip -d ~/searchguard
Download the
search-guard.yml
configuration file. Choose eitherWazuh single-node cluster
if there is only one Wazuh server, orWazuh multi-node cluster
in case of having more than one Wazuh servers:# curl -so ~/searchguard/search-guard.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/searchguard/multi-node/search-guard.yml
After downloading the configuration file in
~/searchguard/search-guard.yml
, replace the values<elasticsearch_X_IP>
and<kibana_ip>
with the corresponding IP addresses. More than one IP can be specified (one entry per line):# Nodes certificates nodes: - name: node-1 dn: CN=node-1,OU=Docu,O=Wazuh,L=California,C=US ip: - <elasticsearch_1_IP> - name: node-2 dn: CN=node-2,OU=Docu,O=Wazuh,L=California,C=US ip: - <elasticsearch_2_IP> - name: node-3 dn: CN=node-3,OU=Docu,O=Wazuh,L=California,C=US ip: - <elasticsearch_3_IP> - name: kibana dn: CN=kibana,OU=Docu,O=Wazuh,L=California,C=US ip: - <kibana_ip>
# curl -so ~/searchguard/search-guard.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/searchguard/multi-node/search-guard-multi-node.yml
After downloading the configuration file, replace the values
<elasticsearch_X_IP>
and<kibana_ip>
with the corresponding IP addresses in the file~/searchguard/search-guard.yml
. More than one IP can be specified (one entry per line):# Nodes certificates nodes: - name: node-1 dn: CN=node-1,OU=Docu,O=Wazuh,L=California,C=US ip: - <elasticsearch_1_IP> - name: node-2 dn: CN=node-2,OU=Docu,O=Wazuh,L=California,C=US ip: - <elasticsearch_2_IP> - name: node-3 dn: CN=node-3,OU=Docu,O=Wazuh,L=California,C=US ip: - <elasticsearch_3_IP> - name: kibana dn: CN=kibana,OU=Docu,O=Wazuh,L=California,C=US ip: - <kibana_ip>
There should as many
filebeat-X
sections as Wazuh servers in the installation:- name: filebeat-1 dn: CN=filebeat-1,OU=Docu,O=Wazuh,L=California,C=US - name: filebeat-2 dn: CN=filebeat-2,OU=Docu,O=Wazuh,L=California,C=US
To learn more about how to create and configure the certificates visit the certificates deployment section.
Run the Search Guard's script to create the certificates:
# ~/searchguard/tools/sgtlstool.sh -c ~/searchguard/search-guard.yml -ca -crt -t /etc/elasticsearch/certs/ # mv /etc/elasticsearch/certs/node-1.pem /etc/elasticsearch/certs/elasticsearch.pem # mv /etc/elasticsearch/certs/node-1.key /etc/elasticsearch/certs/elasticsearch.key # mv /etc/elasticsearch/certs/node-1_http.pem /etc/elasticsearch/certs/elasticsearch_http.pem # mv /etc/elasticsearch/certs/node-1_http.key /etc/elasticsearch/certs/elasticsearch_http.key
In case of further certificates deployments, it is highly recommended to keep Search Guard's TLS offline tool and its configuration file
search-guard.yml
on the master node.Compress all the necessary files to be deployed to all the servers in the distributed deployment:
# tar -cf /etc/elasticsearch/certs/certs.tar *
Copy
certs.tar
to all the instances of the distributed deployment. This can be done by using, for example,scp
.Remove unnecessary files:
# rm /etc/elasticsearch/certs/client-certificates.readme /etc/elasticsearch/certs/elasticsearch_elasticsearch_config_snippet.yml search-guard-tlstool-1.7.zip filebeat* node-* -f
If Kibana will be installed in this node, keep the certificates file. Otherwise, if the file has been already copied to all the instances of the distributed deployment, remove it to increase security
rm -f certs.tar
.
Enable and start the Elasticsearch service:
# systemctl daemon-reload # systemctl enable elasticsearch # systemctl start elasticsearch
Choose one option according to the OS used:
RPM based OS:
# chkconfig --add elasticsearch # service elasticsearch start
Debian based OS:
# update-rc.d elasticsearch defaults 95 10 # service elasticsearch start
Note
The Open Distro for Elasticsearch performance analyzer plugin is installed by default and can have a negative impact on system resources. We recommend removing it with the following command /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_performance_analyzer
. Please be sure to restart the Elasticsearch service afterwards.
Subsequent nodes
The following steps should be executed in each of the subsequent nodes of the Elasticsearch cluster.
Prerequisites
Open Distro for Elasticsearch requires the Java Development Kit and other packages installation including wget
, curl
, and unzip
that will be used in further steps:
Install all the necessary packages for the installation:
# export JAVA_HOME=/usr/ && yum install curl unzip wget && yum install java-11-openjdk-devel
In case JDK 11 is not available for the operating system being used, install the package adoptopenjdk-11-hotspot
using Adopt Open JDK.
Install all the necessary packages for the installation:
# apt install curl apt-transport-https unzip wget software-properties-common
Add the repository for Java Development Kit (JDK):
For Debian:
# echo 'deb http://deb.debian.org/debian stretch-backports main' > /etc/apt/sources.list.d/backports.listFor Ubuntu and other Debian based OS:
# add-apt-repository ppa:openjdk-r/ppa
Update repository data:
# apt update
Install all the required utilities:
# export JAVA_HOME=/usr/ && apt install openjdk-11-jdk
In case JDK 11 is not available for the operating system being used, install the package adoptopenjdk-11-hotspot
using Adopt Open JDK.
Install all the necessary packages for the installation:
# export JAVA_HOME=/usr/ && zypper install curl unzip wget && zypper install java-11-openjdk-devel
In case JDK 11 is not available for the operating system being used, install the package adoptopenjdk-11-hotspot
using Adopt Open JDK.
Adding the Wazuh repository
Import the GPG key:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
Add the repository:
# cat > /etc/yum.repos.d/wazuh.repo << EOF [wazuh] gpgcheck=1 gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH enabled=1 name=EL-$releasever - Wazuh baseurl=https://packages.wazuh.com/4.x/yum/ protect=1 EOF
Install the GPG key:
# curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
Add the repository:
# echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
Update the package information:
# apt-get update
Import the GPG key:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
Add the repository:
# cat > /etc/zypp/repos.d/wazuh.repo <<\EOF [wazuh] gpgcheck=1 gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH enabled=1 name=EL-$releasever - Wazuh baseurl=https://packages.wazuh.com/4.x/yum/ protect=1 EOF
Elasticsearch installation
Install Open Distro for Elasticsearch:
Install Open Distro for Elasticsearch:
# yum install opendistroforelasticsearch-1.11.0-1
Install Elasticsearch OSS and Open Distro for Elasticsearch:
# apt install elasticsearch-oss=7.9.1 opendistro-alerting=1.11.0.1-1 opendistro-anomaly-detection=1.11.0.0-1 opendistro-index-management=1.11.0.0-1 opendistro-job-scheduler=1.11.0.0-1 opendistro-knn=1.11.0.0-1 opendistro-knnlib=1.11.0.0 opendistro-performance-analyzer=1.11.0.0-1 opendistro-security=1.11.0.0-0 opendistro-sql=1.11.0.0-1 opendistroforelasticsearch=1.11.0-1
Install Open Distro for Elasticsearch:
# zypper install opendistroforelasticsearch-1.11.0-1
Elasticsearch configuration
To configure Elasticsearch download the following file /etc/elasticsearch/elasticsearch.yml
:
# curl -so /etc/elasticsearch/elasticsearch.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/elasticsearch/7.x/elasticsearch_cluster_subsequent_nodes.yml
Edit the file /etc/elasticsearch/elasticsearch.yml
. By default, the pre-configured template includes 3 nodes, in case of having a different number, lines must be added or removed accordingly:
network.host: <elasticsearch_ip>
node.name: <node-X>
cluster.name: <elastic_cluster>
cluster.initial_master_nodes:
- node-1
- node-2
- node-3
discovery.seed_hosts:
- <elasticsearch_ip_node1>
- <elasticsearch_ip_node2>
- <elasticsearch_ip_node3>
Depending on the node type, some parameters may vary between nodes. The cluster.initial_master_nodes
and the discovery.seed_hosts
are lists of all the master-eligible nodes in the cluster. The parameter node.master: false
must be included in every Elasticsearch node that will not be configured as master.
Values to be replaced:
<elasticsearch_ip>
: the host's IP. E.g.:10.0.0.2
.<node_name>
: The node name. E.g.:node-2
.<elastic_cluster>
: Elasticsearch cluster name. E.g.:elastic-cluster-production
.<elasticsearch_ip_nodeX>
Elasticsearch cluster master-eligible nodes IP. E.g.:10.0.0.3
.The node certificates for each node must be specified under the
opendistro_security.nodes_dn
section.opendistro_security.nodes_dn: - CN=node-1,OU=Docu,O=Wazuh,L=California,C=US - CN=node-2,OU=Docu,O=Wazuh,L=California,C=US - CN=node-3,OU=Docu,O=Wazuh,L=California,C=US - CN=<common_name>,OU=<operational_unit>,O=<organization_name>,L=<locality>,C=<country_code>
Certificates deployment
Remove the demo certificates:
# rm /etc/elasticsearch/esnode-key.pem /etc/elasticsearch/esnode.pem /etc/elasticsearch/kirk-key.pem /etc/elasticsearch/kirk.pem /etc/elasticsearch/root-ca.pem -f
Configure Elasticsearch certificates:
The next step is the certificates placement, replace the
X
with the corresponding node number and execute the following commands. This guide assumes that the filecerts.tar
is placed in ~/ (root home user folder).
# mkdir /etc/elasticsearch/certs # mv ~/certs.tar /etc/elasticsearch/certs/ # cd /etc/elasticsearch/certs/ # tar -xf certs.tar node-X.pem node-X.key node-X_http.pem node-X_http.key root-ca.pem # mv /etc/elasticsearch/certs/node-X.pem /etc/elasticsearch/certs/elasticsearch.pem # mv /etc/elasticsearch/certs/node-X.key /etc/elasticsearch/certs/elasticsearch.key # mv /etc/elasticsearch/certs/node-X_http.pem /etc/elasticsearch/certs/elasticsearch_http.pem # mv /etc/elasticsearch/certs/node-X_http.key /etc/elasticsearch/certs/elasticsearch_http.key
Remove unnecessary files:
# rm /etc/elasticsearch/certs/client-certificates.readme /etc/elasticsearch/certs/elasticsearch_elasticsearch_config_snippet.yml search-guard-tlstool-1.7.zip filebeat* node-* -f
If Kibana will be installed in this node, keep the certificates file. Otherwise, remove it to increase security
rm -f certs.tar
.
Enable and start the Elasticsearch service:
# systemctl daemon-reload # systemctl enable elasticsearch # systemctl start elasticsearch
Choose one option according to the OS used:
RPM based OS:
# chkconfig --add elasticsearch # service elasticsearch start
Debian based OS:
# update-rc.d elasticsearch defaults 95 10 # service elasticsearch start
Note
The Open Distro for Elasticsearch performance analyzer plugin is installed by default and can have a negative impact on system resources. We recommend removing it with the following command /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_performance_analyzer
. Please be sure to restart the Elasticsearch service afterwards.
Initializing the cluster
Once the installation process is done in all the servers of the Elasticsearch cluster, run the security admin script on the initial node:
Run the Elasticsearch's securityadmin
script to load the new certificates information and start the cluster. To run this command, the value <elasticsearch_IP>
must be replaced by the Elasticsearch installation IP:
# /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -icl -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin.key -h <elasticsearch_IP>
Replace the value <elasticsearch_IP>
by the Elasticsearch installation IP and run the following command to ensure that the installation has been successful:
# curl -XGET https://<elasticsearch_ip>:9200 -u admin:admin -k
An example response should look as follows:
{
"name" : "node-1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "2gIeOOeUQh25c2yU0Pd-RQ",
"version" : {
"number" : "7.9.1",
"build_flavor" : "oss",
"build_type" : "rpm",
"build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
"build_date" : "2020-09-01T21:22:21.964974Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
To verify the nodes that are connected to the cluster, replace <elasticsearch_ip>
and run the following command:
# curl -XGET https://<elasticsearch_ip>:9200/_cat/nodes -u admin:admin -k
It is highly recommended to change Elasticsearch’s default passwords for the users found at the /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
file. More information about this process can be found here. It is also recommended to customize the file /etc/elasticsearch/jvm.options
in order to improve the performance of Elasticsearch. Learn more about this process in the Elasticsearch tuning section.
To uninstall Elasticsearch, visit the uninstalling section.
Next steps
The next step is the installation of the Wazuh server, select the cluster mode: