Elasticsearch multi-node cluster

Open Distro for Elasticsearch is an open source distribution of Elasticsearch, a highly scalable full-text search engine. It offers advanced security, alerting, index management, deep performance analysis, and several other additional features.

You can install Elasticsearch on a multi-node cluster. For security and stability reasons in case Elasticsearch nodes become unavailable, it is recommended to have an odd number of master eligible nodes. Make sure that you take this into consideration when deciding the configuration of your Elasticsearch cluster.

The installation process for a multi-node cluster is performed in three stages.

  1. Initial node configuration

  2. Subsequent nodes configuration

  3. Initializing the Elasticsearch cluster

Note

Root user privileges are necessary to run all the commands.

1. Initial node configuration

The first stage of the process for installing a multi-node cluster is performed on the first Elasticsearch node to be configured. The SSL certificates generated in this stage are later deployed to the subsequent nodes.

Adding the Wazuh repository

  1. Install the necessary packages for the installation:

    # yum install curl unzip wget
    
  2. Import the GPG key:

    # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
    
  3. Add the repository:

    # cat > /etc/yum.repos.d/wazuh.repo << EOF
    [wazuh]
    gpgcheck=1
    gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
    enabled=1
    name=EL-\$releasever - Wazuh
    baseurl=https://packages.wazuh.com/4.x/yum/
    protect=1
    EOF
    
  1. Install the necessary packages for the installation:

    # apt install curl apt-transport-https unzip wget software-properties-common
    
  2. Install the GPG key:

    # curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
    
  3. Add the repository:

    # echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
    
  4. Update the package information:

    # apt-get update
    
  1. Install the necessary packages for the installation:

    # zypper install curl unzip wget
    
  2. Import the GPG key:

    # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
    
  3. Add the repository:

    # cat > /etc/zypp/repos.d/wazuh.repo <<\EOF
    [wazuh]
    gpgcheck=1
    gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
    enabled=1
    name=EL-$releasever - Wazuh
    baseurl=https://packages.wazuh.com/4.x/yum/
    protect=1
    EOF
    

Elasticsearch installation

Install Open Distro for Elasticsearch:

Install Open Distro for Elasticsearch:

# yum install opendistroforelasticsearch

Install Elasticsearch OSS and Open Distro for Elasticsearch:

# apt install elasticsearch-oss opendistroforelasticsearch

Install Open Distro for Elasticsearch:

# zypper install opendistroforelasticsearch

Elasticsearch configuration

To configure Elasticsearch successfully, follow these steps:

  1. Download the /etc/elasticsearch/elasticsearch.yml file.

    # curl -so /etc/elasticsearch/elasticsearch.yml https://packages.wazuh.com/resources/4.2/open-distro/elasticsearch/7.x/elasticsearch_cluster_initial_node.yml
    
  2. Edit the file /etc/elasticsearch/elasticsearch.yml. The pre-configured template includes 3 nodes by default; you can add or remove lines according to the number of nodes in your cluster.

    network.host: <elasticsearch_ip>
    node.name: node-1
    cluster.name: <elastic_cluster>
    cluster.initial_master_nodes:
            - node-1
            - node-2
            - node-3
    discovery.seed_hosts:
            - <elasticsearch_ip_node1>
            - <elasticsearch_ip_node2>
            - <elasticsearch_ip_node3>
    

Depending on the node type, some parameters may vary between nodes. The cluster.initial_master_nodes and the discovery.seed_hosts are lists of all the master-eligible nodes in the cluster. The parameter node.master: false must be included in every Elasticsearch node that will not be configured as master.

Values to be replaced in the file:

  • <elasticsearch_ip>: the host’s IP. For example, 10.0.0.2.

  • <elastic_cluster>: Elasticsearch cluster name. For example, elastic-cluster-production.

  • <elasticsearch_ip_nodeX> Elasticsearch cluster master-eligible nodes IP. For example, 10.0.0.3.

  • The node certificates for each Elasticsearch node must be specified under the opendistro_security.nodes_dn section. Make sure to use the same names to create the nodes certificates.

    opendistro_security.nodes_dn:
        - CN=node-1,OU=Docu,O=Wazuh,L=California,C=US
        - CN=node-2,OU=Docu,O=Wazuh,L=California,C=US
        - CN=node-3,OU=Docu,O=Wazuh,L=California,C=US
        - CN=<common_name>,OU=<operational_unit>,O=<organization_name>,L=<locality>,C=<country_code>
    

Elasticsearch roles and users

To use the Wazuh Kibana plugin properly, run the following command to add new users and roles in Kibana.

# curl -so /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml https://packages.wazuh.com/resources/4.2/open-distro/elasticsearch/roles/roles.yml
# curl -so /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml https://packages.wazuh.com/resources/4.2/open-distro/elasticsearch/roles/roles_mapping.yml
# curl -so /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml https://packages.wazuh.com/resources/4.2/open-distro/elasticsearch/roles/internal_users.yml

Wazuh users added in Kibana by running the commands above:

wazuh_user

It is created for users who need read-only access to the Wazuh Kibana plugin.

wazuh_admin

It is recommended for users who need administrative privileges.

Wazuh additional roles added in Kibana to give the appropriate permissions to users:

wazuh_ui_user

It provides wazuh_user with permissions to read the Wazuh indices.

wazuh_ui_admin

It allows wazuh_admin to perform reading, writing, management, and indexing tasks on the Wazuh indices.

These users and roles are designed to operate along with the Wazuh Kibana plugin, but they are protected and cannot be modified from the Kibana interface. To modify them or add new users or roles, the securityadmin script has to be run.

Certificates creation and deployment

  1. Remove the demo certificates:

    # rm /etc/elasticsearch/esnode-key.pem /etc/elasticsearch/esnode.pem /etc/elasticsearch/kirk-key.pem /etc/elasticsearch/kirk.pem /etc/elasticsearch/root-ca.pem -f
    
  2. Generate and deploy the certificates:

    1. Download the wazuh-cert-tool.sh to create the certificates:

      # curl -so ~/wazuh-cert-tool.sh https://packages.wazuh.com/resources/4.2/open-distro/tools/certificate-utility/wazuh-cert-tool.sh
      # curl -so ~/instances.yml https://packages.wazuh.com/resources/4.2/open-distro/tools/certificate-utility/instances.yml
      
    2. Edit ~/instances.yml and replace the values <node-name> and node-IP with the corresponding names and IP addresses. Add as many nodes fields as needed:

      # Elasticsearch nodes
      elasticsearch-nodes:
        - name: <node-name>
          ip:
            - node-IP
        - name: <node-name>
          ip:
            - node-IP
        - name: <node-name>
          ip:
            - node-IP
      
      # Wazuh server nodes
      wazuh-servers:
        - name: <node-name>
          ip:
            - node-IP
        - name: <node-name>
          ip:
            - node-IP
      
      # Kibana node
      kibana:
        - name: <node-name>
          ip:
            - node-IP
      

      To learn more about how to create and configure the certificates, see the Certificates deployment section.

    3. Run the wazuh-cert-tool.sh to create the certificates:

      #  bash ~/wazuh-cert-tool.sh
      
    4. Replace elasticsearch-node-name with your Elasticsearch node name, the same used in instances.yml to create the certificates, and move the certificates to their corresponding location:

      # node_name=elasticsearch-node-name
      
      # mkdir /etc/elasticsearch/certs/
      # mv ~/certs/$node_name* /etc/elasticsearch/certs/
      # mv ~/certs/admin* /etc/elasticsearch/certs/
      # cp ~/certs/root-ca* /etc/elasticsearch/certs/
      # mv /etc/elasticsearch/certs/$node_name.pem /etc/elasticsearch/certs/elasticsearch.pem
      # mv /etc/elasticsearch/certs/$node_name-key.pem /etc/elasticsearch/certs/elasticsearch-key.pem
      
    5. Compress all the necessary files to be sent to all the instances:

      # cd ~/certs/
      # tar -cvf certs.tar *
      # mv ~/certs/certs.tar ~/
      
    6. Copy certs.tar to all the servers of the distributed deployment. This can be done by using, for example, scp.

    7. If Kibana will be installed on this node, keep the certificates file. Otherwise, if the file is already copied to all the instances of the distributed deployment, remove it to increase security rm -f certs.tar.

  3. Enable and start the Elasticsearch service:

    # systemctl daemon-reload
    # systemctl enable elasticsearch
    # systemctl start elasticsearch
    

    Choose one option according to the operating system used:

    1. RPM based operating system:

    # chkconfig --add elasticsearch
    # service elasticsearch start
    
    1. Debian based operating system:

    # update-rc.d elasticsearch defaults 95 10
    # service elasticsearch start
    

Note

The Open Distro for Elasticsearch performance analyzer plugin is installed by default and can have a negative impact on system resources. We recommend removing it with the following command /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro-performance-analyzer. Make sure to restart the Elasticsearch service afterwards.

2. Subsequent nodes configuration

The second stage of the process for installing a multi-node cluster is performed on the subsequent nodes of the Elasticsearch cluster.

Adding the Wazuh repository

  1. Install the necessary packages for the installation:

    # yum install curl unzip wget
    
  2. Import the GPG key:

    # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
    
  3. Add the repository:

    # cat > /etc/yum.repos.d/wazuh.repo << EOF
    [wazuh]
    gpgcheck=1
    gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
    enabled=1
    name=EL-\$releasever - Wazuh
    baseurl=https://packages.wazuh.com/4.x/yum/
    protect=1
    EOF
    
  1. Install the necessary packages for the installation:

    # apt install curl apt-transport-https unzip wget software-properties-common
    
  2. Install the GPG key:

    # curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
    
  3. Add the repository:

    # echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
    
  4. Update the package information:

    # apt-get update
    
  1. Install the necessary packages for the installation:

    # zypper install curl unzip wget
    
  2. Import the GPG key:

    # rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
    
  3. Add the repository:

    # cat > /etc/zypp/repos.d/wazuh.repo <<\EOF
    [wazuh]
    gpgcheck=1
    gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
    enabled=1
    name=EL-$releasever - Wazuh
    baseurl=https://packages.wazuh.com/4.x/yum/
    protect=1
    EOF
    

Elasticsearch installation

Install Open Distro for Elasticsearch:

Install Open Distro for Elasticsearch:

# yum install opendistroforelasticsearch

Install Elasticsearch OSS and Open Distro for Elasticsearch:

# apt install elasticsearch-oss opendistroforelasticsearch

Install Open Distro for Elasticsearch:

# zypper install opendistroforelasticsearch

Elasticsearch configuration

To configure Elasticsearch successfully, follow these steps:

  1. Download the /etc/elasticsearch/elasticsearch.yml file.

    # curl -so /etc/elasticsearch/elasticsearch.yml https://packages.wazuh.com/resources/4.2/open-distro/elasticsearch/7.x/elasticsearch_cluster_subsequent_nodes.yml
    
  2. Edit the file /etc/elasticsearch/elasticsearch.yml. The pre-configured template includes 3 nodes by default; you can add or remove lines according to the number of nodes in your cluster.

    network.host: <elasticsearch_ip>
    node.name: <node-X>
    cluster.name: <elastic_cluster>
    cluster.initial_master_nodes:
            - node-1
            - node-2
            - node-3
    discovery.seed_hosts:
            - <elasticsearch_ip_node1>
            - <elasticsearch_ip_node2>
            - <elasticsearch_ip_node3>
    

    Depending on the node type, some parameters may vary between nodes. The cluster.initial_master_nodes and the discovery.seed_hosts are lists of all the master-eligible nodes in the cluster. The parameter node.master: false must be included in every Elasticsearch node that will not be configured as master.

    Values to be replaced in the file:

    • <elasticsearch_ip>: the host’s IP. For example, 10.0.0.2.

    • <node_name>: The node name. For example, node-2.

    • <elastic_cluster>: Elasticsearch cluster name. For example, elastic-cluster-production.

    • <elasticsearch_ip_nodeX> Elasticsearch cluster master-eligible nodes IP. For example, 10.0.0.3.

    • The node certificates for each node must be specified under the opendistro_security.nodes_dn section.

      opendistro_security.nodes_dn:
          - CN=node-1,OU=Docu,O=Wazuh,L=California,C=US
          - CN=node-2,OU=Docu,O=Wazuh,L=California,C=US
          - CN=node-3,OU=Docu,O=Wazuh,L=California,C=US
          - CN=<common_name>,OU=<operational_unit>,O=<organization_name>,L=<locality>,C=<country_code>
      

Certificates deployment

  1. Remove the demo certificates:

    # rm /etc/elasticsearch/esnode-key.pem /etc/elasticsearch/esnode.pem /etc/elasticsearch/kirk-key.pem /etc/elasticsearch/kirk.pem /etc/elasticsearch/root-ca.pem -f
    
  2. Configure Elasticsearch certificates:

    1. Replace elasticsearch-node-name with your Elasticsearch node name, the same used in instances.yml to create the certificates, and move the certificates to their corresponding location. The certs.tar file should be placed in ~/ (root home user folder).

      # node_name=elasticsearch-node-name
      
      # mkdir /etc/elasticsearch/certs
      # mv ~/certs.tar /etc/elasticsearch/certs/
      # cd /etc/elasticsearch/certs/
      # tar -xf certs.tar $node_name.pem $node_name-key.pem root-ca.pem
      # mv /etc/elasticsearch/certs/$node_name.pem /etc/elasticsearch/certs/elasticsearch.pem
      # mv /etc/elasticsearch/certs/$node_name-key.pem /etc/elasticsearch/certs/elasticsearch-key.pem
      
    2. If Kibana will be installed in this node, keep the certificates file. Otherwise, remove it to increase security rm -f certs.tar.

  3. Enable and start the Elasticsearch service:

    # systemctl daemon-reload
    # systemctl enable elasticsearch
    # systemctl start elasticsearch
    

    Choose one option according to the operating system used:

    1. RPM based operating system:

    # chkconfig --add elasticsearch
    # service elasticsearch start
    
    1. Debian based operating system:

    # update-rc.d elasticsearch defaults 95 10
    # service elasticsearch start
    

Note

The Open Distro for Elasticsearch performance analyzer plugin is installed by default and can have a negative impact on system resources. We recommend removing it with the following command /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro-performance-analyzer. Make sure to restart the Elasticsearch service afterward.

3. Initializing the cluster

The final stage of the process for installing a multi-node cluster is performed on the initial node and consists in running the security admin script.

  1. Run the Elasticsearch securityadmin script to load the new certificates information and start the cluster. To run this command, the value <elasticsearch_IP> must be replaced with the Elasticsearch installation IP:

    # export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -icl -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem -h <elasticsearch_IP>
    
  2. To confirm that the installation is successful, run the following command replacing the value <elasticsearch_IP> with the Elasticsearch installation IP.

    # curl -XGET https://<elasticsearch_ip>:9200 -u admin:admin -k
    

    An example response should look as follows:

    {
      "name" : "node-1",
      "cluster_name" : "elasticsearch",
      "cluster_uuid" : "tWYgqpgdRz6fGN8gH11flw",
      "version" : {
        "number" : "7.10.2",
        "build_flavor" : "oss",
        "build_type" : "rpm",
        "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
        "build_date" : "2021-01-13T00:42:12.435326Z",
        "build_snapshot" : false,
        "lucene_version" : "8.7.0",
        "minimum_wire_compatibility_version" : "6.8.0",
        "minimum_index_compatibility_version" : "6.0.0-beta1"
      },
      "tagline" : "You Know, for Search"
    }
    

Note

It is highly recommended to change the default passwords of Elasticsearch for the users’ passwords. To perform this action, see the Elasticsearch tuning section.

It is also recommended to customize the file /etc/elasticsearch/jvm.options in order to improve the performance of Elasticsearch. Learn more about these processes in the user manual.

To uninstall Elasticsearch, visit the Uninstalling section.

Next steps

The next step is the installation of the Wazuh server. First, choose the cluster mode.