Elasticsearch multi-node cluster

Elasticsearch is a highly scalable full-text search and analytics engine. This document will explain how to install the Elastic Stack components in a multi-node cluster.

For resilience in case Elasticsearch nodes become unavailable, it is recommended to have an odd number of master eligible nodes, please take this into consideration when deciding the configuration of your Elasticsearch cluster.

Note

Root user privileges are necessary to execute all the commands described below.

Installing Elasticsearch

The installation process for a multi-node cluster will be explained in three parts. The first one refers to the configuration of the initial node, in which the certificates that will be deployed to the subsequent nodes are generated.

The second part will explain how to configure the remaining nodes of the cluster. Finally, the third part provides instructions for initializing the Elasticsearch cluster and verifying that everything is working properly.

Initial node

The following instructions are meant to be performed on the first Elasticsearch node to be configured.

Prerequisites

Install all the necessary packages:

# yum install zip unzip curl

Install all the necessary packages:

# apt-get install lsb-release curl apt-transport-https zip unzip gnupg2

Install all the necessary packages:

# zypper install zip unzip curl

Adding the Elastic Stack repository

The addition of Elastic Stack repository must be done in all Elasticsearch cluster nodes.

  1. Import the GPG key:

    # rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
    
  2. Add the repository:

    # cat > /etc/yum.repos.d/elastic.repo << EOF
    [elasticsearch-7.x]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    EOF
    
  1. Install the GPG key:

    # curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
    
  2. Add the repository:

    # echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list
    
  3. Update the package information:

    # apt-get update
    
  1. Import the GPG key:

    # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    
  2. Add the repository:

    # cat > /etc/zypp/repos.d/elastic.repo <<\EOF
    [elasticsearch-7.x]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    EOF
    

Elasticsearch installation and configuration

Install the Elasticsearch package:

# yum install elasticsearch-7.9.3
# apt-get install elasticsearch=7.9.3
# zypper install elasticsearch-7.9.3

Once Elasticsearch is installed it has to be configured by downloading and editing the file /etc/elasticsearch/elasticsearch.yml as follows:

# curl -so /etc/elasticsearch/elasticsearch.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/elastic-stack/elasticsearch/7.x/elasticsearch_cluster_initial_node.yml

The file /etc/elasticsearch/elasticsearch.yml has to be edited:

network.host: <elasticsearch_ip>
node.name: elasticsearch-1
cluster.name: elasticsearch_cluster
cluster.initial_master_nodes:
        - elasticsearch-1
        - elasticsearch-2
        - elasticsearch-3
discovery.seed_hosts:
        - <elasticsearch_ip_node1>
        - <elasticsearch_ip_node2>
        - <elasticsearch_ip_node3>

Depending on the node type, some parameters may vary between nodes. cluster.initial_master_nodes and discovery.seed_hosts are lists of all the master-eligible nodes in the cluster. The parameter node.master: false must be included in every Elasticsearch node that will not be configured as master.

Values to be replaced:

  • <elasticsearch_ip>: the host’s IP. E.g.: 10.0.0.2.

  • <elasticsearch_ip_nodeX> Elasticsearch cluster master-eligible nodes IP. E.g.: 10.0.0.3.

Certificates creation and deployment

  1. This step implies the selection of the Wazuh cluster mode. Choose between Wazuh single-node cluster, if having only one Wazuh server, and Wazuh multi-node cluster in case of having two or more Wazuh servers.

    The instances file /usr/share/elasticsearch/instances.yml can be created as follows:

    cat > /usr/share/elasticsearch/instances.yml <<\EOF
    instances:
    - name: "elasticsearch-1"
      ip:
      - "10.0.0.2"
    - name: "elasticsearch-2"
      ip:
      - "10.0.0.3"
    - name: "elasticsearch-3"
      ip:
      - "10.0.0.4"
    - name: "filebeat"
      ip:
      - "10.0.0.5"
    - name: "kibana"
      ip:
      - "10.0.0.6"
    EOF
    

    Every name section corresponds to one host in the Wazuh - Elastic Stack environment. In this example, the file describes:

    • A filebeat instance with IP 10.0.0.5. It is a Wazuh single-node cluster.

    • Three elasticsearch instances, #1, #2 and #3 with their respective IPs 10.0.0.2, 10.0.0.3 and 10.0.0.4. All belong to three Elasticsearch cluster nodes. In case of configuring an Elasticsearch multi-node cluster with four or more nodes, more name sections can be defined with their respective names and IPs.

    • A kibana instance with IP 10.0.0.6. If Kibana will be installed in the same server as Elasticsearch, the same IP may be used.

    Replace the IPs with the corresponding addresses for each instance in your environment.

    Create the certificates using the elasticsearch-certutil tool:

    # /usr/share/elasticsearch/bin/elasticsearch-certutil cert ca --pem --in instances.yml --keep-ca-key --out ~/certs.zip
    

    The resulting file certs.zip contains a directory for each instance included in instances.yml. Each directory contains a certificate and a private key necessary to secure communications.

    The instances file /usr/share/elasticsearch/instances.yml can be created as follows:

    cat > /usr/share/elasticsearch/instances.yml <<\EOF
    instances:
    - name: "elasticsearch-1"
      ip:
      - "10.0.0.2"
    - name: "elasticsearch-2"
      ip:
      - "10.0.0.3"
    - name: "elasticsearch-3"
      ip:
      - "10.0.0.4"
    - name: "filebeat-1"
      ip:
      - "10.0.0.5"
    - name: "filebeat-2"
      ip:
      - "10.0.0.6"
    - name: "kibana"
      ip:
      - "10.0.0.7"
    EOF
    

    Every name section corresponds one host in the Wazuh - Elastic Stack environment. In this example, the file describes:

    • Two filebeat instances, #1 and #2 with their respective IPs 10.0.0.5 and 10.0.0.6. Both belong to individual Wazuh cluster nodes. If you want to configure a Wazuh multi-node cluster with three or more nodes, you must define more name sections with their respective names and IPs.

    • Three instances elasticsearch, #1, #2 and #3 with their respective IPs 10.0.0.2, 10.0.0.3 and 10.0.0.4. They belong to three Elasticsearch cluster nodes. In the case of configuring an Elasticsearch multi-node cluster with four or more nodes, more name sections can be defined with their respective names and IPs.

    • A kibana instance with IP 10.0.0.7. If Kibana will be installed in the same server as Elasticsearch, the same IP may be used.

    Replace the IPs with the corresponding addresses for each instance in your environment.

    Create the certificates using the elasticsearch-certutil tool:

    # /usr/share/elasticsearch/bin/elasticsearch-certutil cert ca --pem --in instances.yml --keep-ca-key --out ~/certs.zip
    

    The resulting file certs.zip contains a directory for each instance included in instances.yml. Each directory contains a certificate and a private key necessary to secure communications.

  2. Copy ~/certs.zip to all the servers of the distributed deployment. This can be done by using, for example, scp..

  3. The next step is to create the directory /etc/elasticsearch/certs, and then copy the CA file, the certificate and the key there.

    # unzip ~/certs.zip -d ~/certs
    # mkdir /etc/elasticsearch/certs/ca -p
    # cp -R ~/certs/ca/ ~/certs/elasticsearch-1/* /etc/elasticsearch/certs/
    # mv /etc/elasticsearch/certs/elasticsearch-1.crt /etc/elasticsearch/certs/elasticsearch.crt
    # mv /etc/elasticsearch/certs/elasticsearch-1.key /etc/elasticsearch/certs/elasticsearch.key
    # chown -R elasticsearch: /etc/elasticsearch/certs
    # chmod -R 500 /etc/elasticsearch/certs
    # chmod 400 /etc/elasticsearch/certs/ca/ca.* /etc/elasticsearch/certs/elasticsearch.*
    # rm -rf ~/certs
    
  4. If Kibana will be installed in this node, keep the certificates file. Otherwise, if the file has been copied already to all the instances of the distributed deployment, remove it to increase security rm -f ~/certs.zip.

  5. Enable and start the Elasticsearch service:

    # systemctl daemon-reload
    # systemctl enable elasticsearch
    # systemctl start elasticsearch
    

    Choose one option according to the OS used:

    1. Debian based OS

    # update-rc.d elasticsearch defaults 95 10
    # service elasticsearch start
    
    1. RPM based OS

    # chkconfig --add elasticsearch
    # service elasticsearch start
    

Subsequent nodes

The following steps should be executed in each of the subsequent nodes of the Elasticsearch cluster.

Prerequisites

Install all the necessary packages:

# yum install zip unzip curl

Install all the necessary packages:

# apt-get install lsb-release curl apt-transport-https zip unzip gnupg2

Install all the necessary packages:

# zypper install zip unzip curl

Adding the Elastic Stack repository

The addition of Elastic Stack repository must be done in all Elasticsearch cluster nodes.

  1. Import the GPG key:

    # rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
    
  2. Add the repository:

    # cat > /etc/yum.repos.d/elastic.repo << EOF
    [elasticsearch-7.x]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    EOF
    
  1. Install the GPG key:

    # curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
    
  2. Add the repository:

    # echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list
    
  3. Update the package information:

    # apt-get update
    
  1. Import the GPG key:

    # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    
  2. Add the repository:

    # cat > /etc/zypp/repos.d/elastic.repo <<\EOF
    [elasticsearch-7.x]
    name=Elasticsearch repository for 7.x packages
    baseurl=https://artifacts.elastic.co/packages/7.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    EOF
    

Elasticsearch installation and configuration

  1. Install the Elasticsearch package:

# yum install elasticsearch-7.9.3
# apt-get install elasticsearch=7.9.3
# zypper install elasticsearch-7.9.3

Once Elasticsearch is installed it has to be configured by downloading and editing the file /etc/elasticsearch/elasticsearch.yml as follows:

# curl -so /etc/elasticsearch/elasticsearch.yml https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/elastic-stack/elasticsearch/7.x/elasticsearch_cluster_subsequent_nodes.yml

The file /etc/elasticsearch/elasticsearch.yml has to be edited:

network.host: <elasticsearch_ip>
node.name: <elasticsearch-X>
cluster.name: elasticsearch_cluster
cluster.initial_master_nodes:
        - elasticsearch-1
        - elasticsearch-2
        - elasticsearch-3
discovery.seed_hosts:
        - <elasticsearch_ip_node1>
        - <elasticsearch_ip_node2>
        - <elasticsearch_ip_node3>

Depending on the node type, some parameters may vary between nodes. cluster.initial_master_nodes and discovery.seed_hosts are lists of all the master-eligible nodes in the cluster. The parameter node.master: false must be included in every Elasticsearch node that will not be configured as master.

Values to be replaced:

  • <elasticsearch_ip>: the host’s IP. E.g.: 10.0.0.2.

  • <node_name>: The node name, change the X for your node number. E.g.: elasticsearch-2.

  • <elasticsearch_ip_nodeX> Elasticsearch cluster master-eligible nodes IP. E.g.: 10.0.0.3.

For more information, please see important discovery and cluster formation settings.

Certificates deployment

  1. The next step is to create the directory /etc/elasticsearch/certs, and then copy the CA file, the certificate and the key there. The X must be replaced according to the defined data in instances.yml file:

    # unzip ~/certs.zip -d ~/certs
    # rm -f ~/certs/ca/ca.key
    # mkdir /etc/elasticsearch/certs/ca -p
    # cp -R ~/certs/ca/ ~/certs/elasticsearch-X/* /etc/elasticsearch/certs/
    # mv /etc/elasticsearch/certs/elasticsearch-X.crt /etc/elasticsearch/certs/elasticsearch.crt
    # mv /etc/elasticsearch/certs/elasticsearch-X.key /etc/elasticsearch/certs/elasticsearch.key
    # chown -R elasticsearch: /etc/elasticsearch/certs
    # chmod -R 500 /etc/elasticsearch/certs
    # chmod 400 /etc/elasticsearch/certs/ca/ca.* /etc/elasticsearch/certs/elasticsearch.*
    # rm -rf ~/certs
    
  2. If Kibana will be installed in this node, keep the certificates file. Otherwise, remove it to increase security rm -f ~/certs.zip.

  3. Enable and start the Elasticsearch service:

    # systemctl daemon-reload
    # systemctl enable elasticsearch
    # systemctl start elasticsearch
    

    Choose one option according to the OS used:

    1. Debian based OS

    # update-rc.d elasticsearch defaults 95 10
    # service elasticsearch start
    
    1. RPM based OS

    # chkconfig --add elasticsearch
    # service elasticsearch start
    

Initializing the cluster

Once the installation process is done in all the servers of the Elasticsearch cluster, run the following command on the initial node to generate credentials for all the Elastic Stack pre-built roles and users:

# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

The command above will prompt an output like this. Save the password of the elastic user for further steps:

Changed password for user apm_system
PASSWORD apm_system = lLPZhZkB6oUOzzCrkLSF

Changed password for user kibana_system
PASSWORD kibana_system = TaLqVOnSoqKTYLIU0vDn

Changed password for user kibana
PASSWORD kibana = TaLqVOvXoqKTYLIU0vDn

Changed password for user logstash_system
PASSWORD logstash_system = UtuDv2tWkXGYL83v9kWA

Changed password for user beats_system
PASSWORD beats_system = qZcbvCslafMpoEOrE9Ob

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = LzJpQiSylncmCU2GLBTS

Changed password for user elastic
PASSWORD elastic = AN4UeQGA7HGl5iHpMla7

Disabling repositories

This installation guide describes how to install and configure Wazuh and Elastic Stack by first configuring their repositories.

With each new release of Wazuh or Elastic Stack, the development team at Wazuh thoroughly tests the compatibility of each component and performs necessary adjustments before releasing a new Wazuh Kibana plugin.

We recommend disabling the repositories so that the individual packages will not be updated unintentionally which could potentially lead to having a version of the Elastic Stack for which the Wazuh integration has not been released yet.

# sed -i "s/^enabled=1/enabled=0/" /etc/yum.repos.d/elastic.repo
# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/elastic-7.x.list
# apt-get update
# sed -i "s/^enabled=1/enabled=0/" /etc/zypp/repos.d/elastic.repo

To uninstall Elasticsearch, visit the uninstalling section.

Next steps

The next step is the installation of the Wazuh server, select the cluster mode: