Elasticsearch tuning

Change users' password

Changing the default passwords of Elasticsearch is highly recommended in order to improve security.

# curl -k -X POST -u <username>:<user_password> "https://<elasticsearch_ip>:9200/_security/user/<username>/_password?pretty" -H 'Content-Type: application/json' -d '
# {
#   "password" : "<new_password>"
# }
# '

Memory locking

Elasticsearch malfunctions when the system is swapping memory. It is crucial for the health of the node that none of the JVM is ever swapped out to disk. The following steps show how to set the bootstrap.memory_lock setting to true so Elasticsearch will lock the process address space into RAM. This prevents any Elasticsearch memory from being swapped out.

1. Set bootstrap.memory_lock:

   Uncomment or add this line to the /etc/elasticsearch/elasticsearch.yml file:

   bootstrap.memory_lock: true
   

2. Edit the limit of system resources:

   Where to configure system settings depends on which package and operating system used for the Elasticsearch installation.

   SystemdSysV Init

   In a case where systemd is used, system limits need to be specified via systemd. To do this, create the folder executing the command:

   # mkdir -p /etc/systemd/system/elasticsearch.service.d/
   

   Then, in the new directory, add a file called elasticsearch.conf and specify any changes in that file:

   # cat > /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf << EOF
   [Service]
   LimitMEMLOCK=infinity
   EOF
   

3. Limit memory:

   The previous configuration might cause node instability or even node death with an OutOfMemory exception if Elasticsearch tries to allocate more memory than is available. JVM heap limits will help limit memory usage and prevent this situation. Two rules must be applied when setting Elasticsearch's heap size:

   • Use no more than 50% of available RAM.

   • Use no more than 32 GB.

   It is also important to consider the memory usage of the operating system, services and software running on the host. By default, Elasticsearch is configured with a heap of 1 GB. It can be changed via JVM flags using the /etc/elasticsearch/jvm.options file:

   # Xms represents the initial size of total heap space
   # Xmx represents the maximum size of total heap space
   
   -Xms4g
   -Xmx4g
   

   Warning

   The values min (Xms) and max (Xmx) sizes must be the same to prevent JVM heap resizing at runtime as this is a very costly process.

4. Restart Elasticsearch:

# systemctl daemon-reload
# systemctl restart elasticsearch

# service elasticsearch restart

After starting Elasticsearch, run the following request to verify that the setting was successfully changed by checking the value of mlockall:

# curl "http://localhost:9200/_nodes?filter_path=**.mlockall&pretty"

{
  "nodes" : {
    "sRuGbIQRRfC54wzwIHjJWQ" : {
      "process" : {
        "mlockall" : true
      }
    }
  }
}

If the output of the "mlockall" field is false, the request has failed. In addition, the following line will appear in /var/log/elasticsearch/elasticsearch.log:

Unable to lock JVM Memory

References:

• Memory lock check.

• bootstrap.memory_lock.

• Enable bootstrap.memory_lock.

• Heap: Sizing and Swapping.

• Limiting memory usage.

Shards and replicas

Elasticsearch offers the possibility to split an index into multiple segments called shards. Each shard is in itself a fully functional and independent "index" that can be hosted on any node in the cluster. The splitting is important for two main reasons:

• Horizontal scalation.

• Distribute and parallelize operations across shards, increasing the performance and throughput.

In addition, Elasticsearch allows the user to make one or more copies of the index shards in what are called replica shards, or replicas for short. Replication is important for two main reasons

• It provides high availability in case a shard or node fails.

• It allows search volume and throughput to scale, since searches can be executed on all replicas in parallel.

# curl -X PUT "http://localhost:9200/wazuh-alerts-\*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "settings" : {
    "number_of_replicas" : 0
  }
}'