Wazuh indexer tuning

Number of shards for an index

Before creating the first index, consider carefully how many shards will be needed. It is not possible to change the number of shards without re-indexing.

The number of shards needed for optimal performance depends on the number of nodes in the Wazuh indexer cluster. As a general rule, the number of shards must be the same as the number of nodes. For example, a cluster with three nodes should have three shards, while a cluster with only one node would only need one.

Number of replicas for an index

The number of replicas depends on the available storage for the indices. Here is an example of how a Wazuh indexer cluster with three nodes and three shards could be set up.

• No replica: Each node has one shard. If one node goes down, an incomplete index of only two shards is available.

• One replica: Each node has one shard and one replica. If one node goes down, a full index is still available.

• Two replicas: Each node has the full index with one shard and two replicas. With this setup, the cluster continues to operate even if two nodes go down. Although this seems to be the best solution, it increases the storage requirements.

The image below shows a Wazuh indexer cluster with three nodes, each containing a primary shard and two replica shards.

Setting the number of shards

The default installation of a Wazuh indexer node creates each index with three primary shards and no replicas. You can modify the number of primary shards and replicas by loading a new template using the Wazuh API.

In the following example, we set the number of shards for a single-node Wazuh indexer to 1. Perform the following steps on the Wazuh indexer node or any central component allowed to authenticate using the Wazuh API.

1. Download the Wazuh indexer template:

   # curl https://raw.githubusercontent.com/wazuh/wazuh/v4.9.2/extensions/elasticsearch/7.x/wazuh-template.json -o w-indexer-template.json
   

2. Edit w-indexer-template.json to set index.number_of_shards to 1. To avoid Filebeat overwriting the existing template, set the order to 1. Multiple matching templates in the same order result in a non-deterministic merging order.

   {
     "order": 1,
     "index_patterns": [
       "wazuh-alerts-4.x-*",
       "wazuh-archives-4.x-*"
     ],
     "settings": {
       "index.refresh_interval": "5s",
       "index.number_of_shards": "1",
       "index.number_of_replicas": "0",
       "index.auto_expand_replicas": "0-1",
       "index.mapping.total_fields.limit": 10000,
       ...
   

3. Load the new settings.

   # curl -X PUT "https://<INDEXER_IP_ADDRESS>:9200/_template/wazuh-custom" -H 'Content-Type: application/json' -d @w-indexer-template.json -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD>
   

   {"acknowledged":true}
   

4. Confirm that the configuration was successfully updated.

   # curl "https://<INDEXER_IP_ADDRESS>:9200/_template/wazuh-custom?pretty&filter_path=wazuh-custom.settings" -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD>
   

   {
     "wazuh-custom" : {
       "settings" : {
         "index" : {
           "mapping" : {
             "total_fields" : {
               "limit" : "10000"
             }
           },
           "refresh_interval" : "5s",
           "number_of_shards" : "1",
           "auto_expand_replicas" : "0-1",
           "number_of_replicas" : "0",
           ...
   

If the index has already been created, it must be re-indexed.

Setting the number of replicas

The number of replicas can be changed dynamically using the Wazuh indexer API. In a single-node cluster, the number of replicas should be set to zero. This is accomplished by running the following command on the Wazuh indexer node or any central component allowed to authenticate using the Wazuh API:

curl -k -u "<INDEXER_USERNAME>:<INDEXER_PASSWORD>" -XPUT "https://<INDEXER_IP_ADDRESS>:9200/wazuh-alerts-" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {
      "number_of_replicas": 0
    }
  }
}'

Shard allocation awareness

Shard allocation awareness attempts to spread primary and replica shards across multiple zones. It is used to allocate a replica shard to a zone different from its primary zone.

In the event of node failure within a zone, you can be rest assured that your replica shards are distributed among your remaining zones. This enhances fault tolerance, safeguarding your data against zone failures and individual node failures.

To configure shard allocation awareness, update the cluster settings:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone"
  }
}

You can either use persistent or transient settings. We recommend using the persistent setting because it persists through a cluster reboot. The transient setting does not persist through a cluster reboot.

Forced awareness

Using the forced awareness implies that primary and replica shards are never allocated to the same zone.

To configure forced awareness, specify all the possible values for your zone attributes:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values":["zoneA", "zoneB"]
  }
}

In case there are other zones, add the other zones to the cluster.routing.allocation.awareness.force.zone.values field.

Allocation filtering

This allows a node to be excluded from shard allocation. A common use case is when you want to decommission a node within a zone.

To move shards off a node before decommissioning it, create a filter that excludes the node using its IP address. This will move all shards allocated to that node before it is shut down. You can also use a wildcard * in a situation where there are more than one node within an IP range to be decommissioned.

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._ip": "192.168.0.*"
  }
}

Cluster manager nodes

Cluster manager nodes manage all cluster-wide configurations and modifications, including adding, removing, and allocating shards to nodes, as well as generating and deleting indices and fields.

A distributed consensus technique is used to elect a single cluster-manager node from among the cluster-manager eligible nodes. This cluster-manager node is reelected in the event that the incumbent node fails.

You can specify that a Wazuh indexer node is the cluster manager node, even though this is already done by default.

Set a Wazuh indexer node role to cluster_manager by adding the following configuration to the /etc/wazuh-indexer/opensearch.yml file:

node.roles: [ cluster_manager ]

Data nodes

The data node is responsible for storing and searching data. It performs all data related operations (indexing, searching, aggregating) on local shards. These are the worker nodes of your Wazuh indexer cluster and need more disk space than any other node type.

Set a Wazuh indexer node role as a data node by adding the following configuration to the /etc/wazuh-indexer/opensearch.yml file:

node.roles: [ data, ingest ]

As you add data nodes it is important to keep them balanced between zones. For example, if you have three zones, add a data node for each zone. We recommend using storage and RAM-heavy nodes.

Coordinating nodes

The coordinating node delegates client requests to the shards on the data nodes, collects and aggregates the results into one final result, and sends it back to the Wazuh dashboard.

Every node is a coordinating node by default, however to make a node a dedicated coordinating node, set node.roles to an empty list:

node.roles: []