Elasticsearch tuning

Memory locking

Elasticsearch performs poorly when the system is swapping the memory. It is vitally important to the health of your node that none of the JVM is ever swapped out to disk.

In this guide, we will show how to set the bootstrap.memory_lock setting to true so Elasticsearch will lock the process address space into RAM. This prevents any Elasticsearch memory from being swapped out.

1. Set bootstrap.memory_lock

Uncomment or add this line to the /etc/elasticsearch/elasticsearch.yml file:

bootstrap.memory_lock: true

2. Edit the limit of system resources

Where to configure systems settings depends on which package and operating system you choose to use for the Elasticsearch installation.

• In a case where systemd is used, system limits need to be specified via systemd. To do this, create the folder executing the command:

# mkdir -p /etc/systemd/system/elasticsearch.service.d/

Then, in the new directory, add a file called elasticsearch.conf and specify any changes in that file:

[Service]
LimitMEMLOCK=infinity

In other cases, edit the proper file /etc/sysconfig/elasticsearch for RPM or /etc/default/elasticsearch for Debian:

MAX_LOCKED_MEMORY=unlimited

3. Limit memory

The previous configuration might cause node instability or even node death with an OutOfMemory exception if Elasticsearch tries to allocate more memory than is available. JVM heap limits will help limit the memory usage and prevent this situation.

There are two rules to apply when setting the Elasticsearch heap size:

• Use no more than 50% of available RAM.

• Use no more than 32 GB.

In addition, it is important to take into account the memory usage of the operating system, services and software that are running on the host.

By default, Elasticsearch is configured with a 1 GB heap. You can change the heap size via JVM flags using the /etc/elasticsearch/jvm.options file:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

Warning

Ensure that the min (Xms) and max (Xmx) sizes are the same to prevent JVM heap resizing at runtime as this is a very costly process.

4. Restart Elasticsearch

1. For Systemd:

# systemctl daemon-reload
# systemctl restart elasticsearch

2. For SysV Init:

# service elasticsearch restart

After starting Elasticsearch, you can see whether this setting was successfully applied by checking the value of mlockall in the output of the next request:

# curl "http://localhost:9200/_nodes?filter_path=**.mlockall&pretty"

{
  "nodes" : {
    "sRuGbIQRRfC54wzwIHjJWQ" : {
      "process" : {
        "mlockall" : true
      }
    }
  }
}

If the output of the "mlockall" field is false, the request has failed. You will also find the line Unable to lock JVM Memory in the logs (located at /var/log/elasticsearch/elasticsearch.log).

Reference:

• Memory lock check.

• bootstrap.memory_lock.

• Enable bootstrap.memory_lock.

• Heap: Sizing and Swapping.

• Limiting memory usage.

Shards and replicas

Elasticsearch provides the ability to split an index into multiple segments called shards. Each shard is, in and of itself, a fully-functional and independent "index" that can be hosted on any node in the cluster. Sharding is important for two primary reasons:

• you can horizontally split/scale your content volume, and

• you can distribute and parallelize operations across shards which increases performance and throughput.

Also, Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short. Replication is important for two primary reasons:

• it provides high availability in case a shard or node fails, and

• it allows you to scale out your search volume and throughput, since searches can be executed on all replicas in parallel.

# curl -X PUT "http://localhost:9200/wazuh-alerts-\*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "settings" : {
    "number_of_replicas" : 0
  }
}'