Elasticsearch multi-node cluster

Elasticsearch is a highly scalable full-text search and analytics engine. This document will explain how to install the Elastic Stack components in a multi-node cluster.

For resilience, in case Elasticsearch nodes become unavailable, it is recommended to have an odd number of master eligible nodes, please take this into consideration when deciding the configuration of your Elasticsearch cluster.

Note

You need root user privileges to run all the commands described below.

Installing Elasticsearch

The installation process for a multi-node cluster will be explained in three parts. The first one refers to the configuration of the initial node, in which the certificates that will be deployed to the subsequent nodes are generated.

The second part will explain how to configure the remaining cluster nodes. Finally, the third part provides instructions for initializing the Elasticsearch cluster and verifying that everything is working properly.

Initial node

The following instructions are meant to be performed on the first Elasticsearch node to be configured.

Installing Prerequisites

Install all the necessary packages:

# yum install zip unzip curl

Install all the necessary packages:

# apt-get install lsb-release curl apt-transport-https zip unzip gnupg

Adding the Elastic Stack repository

The addition of the Elastic Stack repository must be done in all Elasticsearch cluster nodes.

Import the GPG key:

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the repository:

# cat > /etc/yum.repos.d/elastic.repo << EOF
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Install the GPG key:

# curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/elasticsearch.gpg --import && chmod 644 /usr/share/keyrings/elasticsearch.gpg

Add the repository:

# echo "deb [signed-by=/usr/share/keyrings/elasticsearch.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list

Update the package information:
# apt-get update

Elasticsearch installation and configuration

Install the Elasticsearch package:

# yum install elasticsearch-7.17.13

# apt-get install elasticsearch=7.17.13

Once Elasticsearch is installed, it has to be configured by downloading and editing the file /etc/elasticsearch/elasticsearch.yml as follows:

# curl -so /etc/elasticsearch/elasticsearch.yml https://packages.wazuh.com/4.5/tpl/elastic-basic/elasticsearch_cluster_initial_node.yml

The file /etc/elasticsearch/elasticsearch.yml has to be edited:

network.host: <elasticsearch_ip>
node.name: elasticsearch-1
cluster.name: elasticsearch_cluster
cluster.initial_master_nodes:
        - elasticsearch-1
        - elasticsearch-2
        - elasticsearch-3
discovery.seed_hosts:
        - <elasticsearch_ip_node1>
        - <elasticsearch_ip_node2>
        - <elasticsearch_ip_node3>

Depending on the node type, some parameters may vary between nodes. cluster.initial_master_nodes and discovery.seed_hosts are lists of all the master-eligible nodes in the cluster. The parameter node.master: false must be included in every Elasticsearch node that will not be configured as master.

Values to be replaced:

<elasticsearch_ip>: the host's IP. E.g.: 10.0.0.2.
<elasticsearch_ip_nodeX> Elasticsearch cluster master-eligible nodes IP. E.g.: 10.0.0.3.

Certificates creation and deployment

The number of Wazuh servers to be implemented will determine the next step. Choose Wazuh single-node cluster, if there is only one Wazuh server, or Wazuh multi-node cluster in case there are two or more Wazuh servers.
The instances file /usr/share/elasticsearch/instances.yml can be created as follows:
cat > /usr/share/elasticsearch/instances.yml <<\EOF instances: - name: "elasticsearch-1" ip: - "10.0.0.2" - name: "elasticsearch-2" ip: - "10.0.0.3" - name: "elasticsearch-3" ip: - "10.0.0.4" - name: "filebeat" ip: - "10.0.0.5" - name: "kibana" ip: - "10.0.0.6" EOF
Every name section corresponds to one host in the Wazuh Server - Elastic Stack environment. In this example, the file describes:
- A filebeat instance with IP address 10.0.0.5. It is a Wazuh single-node cluster.
- Three elasticsearch instances, #1, #2 and #3 with their respective IPs 10.0.0.2, 10.0.0.3 and 10.0.0.4. All belong to three Elasticsearch cluster nodes. In the case of configuring an Elasticsearch multi-node cluster with four or more nodes, more name sections can be defined with their respective names and IPs.
- A kibana instance with IP address 10.0.0.6. If Kibana will be installed in the same server as Elasticsearch, the same IP address may be used.
Replace the IPs with the corresponding addresses for each instance in your environment.

Create the certificates using the elasticsearch-certutil tool:
# /usr/share/elasticsearch/bin/elasticsearch-certutil cert ca --pem --in instances.yml --keep-ca-key --out ~/certs.zip
The resulting file certs.zip contains a directory for each instance included in instances.yml. Each directory contains a certificate and a private key necessary to secure communications.
The instances file /usr/share/elasticsearch/instances.yml can be created as follows:

cat > /usr/share/elasticsearch/instances.yml <<\EOF instances: - name: "elasticsearch-1" ip: - "10.0.0.2" - name: "elasticsearch-2" ip: - "10.0.0.3" - name: "elasticsearch-3" ip: - "10.0.0.4" - name: "filebeat-1" ip: - "10.0.0.5" - name: "filebeat-2" ip: - "10.0.0.6" - name: "kibana" ip: - "10.0.0.7" EOF
Every name section corresponds one host in the Wazuh Server - Elastic Stack environment. In this example, the file describes:
Two filebeat instances, #1 and #2 with their respective IPs 10.0.0.5 and 10.0.0.6. Both belong to individual Wazuh cluster nodes. If you want to configure a Wazuh multi-node cluster with three or more nodes, you must define more name sections with their respective names and IPs.

Three instances elasticsearch, #1, #2 and #3 with their respective IPs 10.0.0.2, 10.0.0.3 and 10.0.0.4. They belong to three Elasticsearch cluster nodes. In the case of configuring an Elasticsearch multi-node cluster with four or more nodes, more name sections can be defined with their respective names and IPs.

A kibana instance with IP address 10.0.0.7. If Kibana will be installed in the same server as Elasticsearch, the same IP address may be used.

Replace the IPs with the corresponding addresses for each instance in your environment.

Create the certificates using the elasticsearch-certutil tool:

# /usr/share/elasticsearch/bin/elasticsearch-certutil cert ca --pem --in instances.yml --keep-ca-key --out ~/certs.zip

The resulting file certs.zip contains a directory for each instance included in instances.yml. Each directory contains a certificate and a private key necessary to secure communications.
Copy ~/certs.zip to all the servers of the distributed deployment. This can be done by using, for example, scp..

The next step is to create the directory /etc/elasticsearch/certs, and then copy the CA file, the certificate and the key there.

# unzip ~/certs.zip -d ~/certs
# mkdir /etc/elasticsearch/certs/ca -p
# cp -R ~/certs/ca/ ~/certs/elasticsearch-1/* /etc/elasticsearch/certs/
# mv /etc/elasticsearch/certs/elasticsearch-1.crt /etc/elasticsearch/certs/elasticsearch.crt
# mv /etc/elasticsearch/certs/elasticsearch-1.key /etc/elasticsearch/certs/elasticsearch.key
# chown -R elasticsearch: /etc/elasticsearch/certs
# chmod -R 500 /etc/elasticsearch/certs
# chmod 400 /etc/elasticsearch/certs/ca/ca.* /etc/elasticsearch/certs/elasticsearch.*
# rm -rf ~/certs

If you are going to install Kibana in this node, keep the certificates file. Otherwise, if the file has been copied already to all the instances of the distributed deployment, remove it to increase security rm -f ~/certs.zip.

Enable and start the Elasticsearch service:

# systemctl daemon-reload
# systemctl enable elasticsearch
# systemctl start elasticsearch

Choose one option according to the OS used:

Debian based OS

# update-rc.d elasticsearch defaults 95 10
# service elasticsearch start

RPM based OS

# chkconfig --add elasticsearch
# service elasticsearch start

Subsequent nodes

The following steps should be executed in each of the subsequent nodes of the Elasticsearch cluster.

Prerequisites

Install all the necessary packages:

# yum install zip unzip curl

Install all the necessary packages:

# apt-get install lsb-release curl apt-transport-https zip unzip gnupg

Adding the Elastic Stack repository

The addition of the Elastic Stack repository must be done in all Elasticsearch cluster nodes.

Import the GPG key:

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the repository:

# cat > /etc/yum.repos.d/elastic.repo << EOF
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Install the GPG key:

# curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/elasticsearch.gpg --import && chmod 644 /usr/share/keyrings/elasticsearch.gpg

Add the repository:

# echo "deb [signed-by=/usr/share/keyrings/elasticsearch.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list

Update the package information:
# apt-get update

Elasticsearch installation and configuration

Install the Elasticsearch package:

# yum install elasticsearch-7.17.13
# apt-get install elasticsearch=7.17.13
Once Elasticsearch is installed, it has to be configured by downloading and editing the file /etc/elasticsearch/elasticsearch.yml as follows:

# curl -so /etc/elasticsearch/elasticsearch.yml https://packages.wazuh.com/4.5/tpl/elastic-basic/elasticsearch_cluster_subsequent_nodes.yml

The file /etc/elasticsearch/elasticsearch.yml has to be edited:

network.host: <elasticsearch_ip>
node.name: <elasticsearch-X>
cluster.name: elasticsearch_cluster
cluster.initial_master_nodes:
        - elasticsearch-1
        - elasticsearch-2
        - elasticsearch-3
discovery.seed_hosts:
        - <elasticsearch_ip_node1>
        - <elasticsearch_ip_node2>
        - <elasticsearch_ip_node3>

Values to be replaced:

<elasticsearch_ip>: the host's IP. E.g.: 10.0.0.2.
<node_name>: The node name, change the X for your node number. E.g.: elasticsearch-2.
<elasticsearch_ip_nodeX> Elasticsearch cluster master-eligible nodes IP. E.g.: 10.0.0.3.

For more information, please see important discovery and cluster formation settings.

Certificates deployment

The next step is to create the directory /etc/elasticsearch/certs, and then copy the CA file, the certificate and the key there. The X must be replaced according to the defined data in instances.yml file:

# unzip ~/certs.zip -d ~/certs
# rm -f ~/certs/ca/ca.key
# mkdir /etc/elasticsearch/certs/ca -p
# cp -R ~/certs/ca/ ~/certs/elasticsearch-X/* /etc/elasticsearch/certs/
# mv /etc/elasticsearch/certs/elasticsearch-X.crt /etc/elasticsearch/certs/elasticsearch.crt
# mv /etc/elasticsearch/certs/elasticsearch-X.key /etc/elasticsearch/certs/elasticsearch.key
# chown -R elasticsearch: /etc/elasticsearch/certs
# chmod -R 500 /etc/elasticsearch/certs
# chmod 400 /etc/elasticsearch/certs/ca/ca.* /etc/elasticsearch/certs/elasticsearch.*
# rm -rf ~/certs

If you are going to install Kibana in this node, keep the certificates file. Otherwise, remove it to increase security rm -f ~/certs.zip.

Enable and start the Elasticsearch service:

# systemctl daemon-reload
# systemctl enable elasticsearch
# systemctl start elasticsearch

Choose one option according to the OS used:

Debian based OS

# update-rc.d elasticsearch defaults 95 10
# service elasticsearch start

RPM based OS

# chkconfig --add elasticsearch
# service elasticsearch start

Initializing the cluster

Once the installation process is done in all the servers of the Elasticsearch cluster, run the following command on the initial node to generate credentials for all the Elastic Stack pre-built roles and users:

# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

The command above will prompt an output like this. Save the password of the elastic user for further steps:

Changed password for user apm_system
PASSWORD apm_system = lLPZhZkB6oUOzzCrkLSF

Changed password for user kibana_system
PASSWORD kibana_system = TaLqVOnSoqKTYLIU0vDn

Changed password for user kibana
PASSWORD kibana = TaLqVOvXoqKTYLIU0vDn

Changed password for user logstash_system
PASSWORD logstash_system = UtuDv2tWkXGYL83v9kWA

Changed password for user beats_system
PASSWORD beats_system = qZcbvCslafMpoEOrE9Ob

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = LzJpQiSylncmCU2GLBTS

Changed password for user elastic
PASSWORD elastic = AN4UeQGA7HGl5iHpMla7

Disabling repositories

This installation guide describes how to install and configure Wazuh and Elastic Stack by first configuring their repositories.

With each new release of Wazuh or Elastic Stack, the development team at Wazuh thoroughly tests the compatibility of each component and performs necessary adjustments before releasing a new Wazuh Kibana plugin.

We recommend disabling the repositories so that the individual packages will not be updated unintentionally, which could potentially lead to having a version of the Elastic Stack for which the Wazuh integration has not been released yet.

# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo

# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/elastic-7.x.list
# apt-get update

To uninstall Elasticsearch, visit the uninstalling section.

Next steps

The next step is the installation of the Wazuh server, select the cluster mode: