Cluster management
You can manage and monitor the Wazuh indexer cluster in two ways: from the Wazuh dashboard console or with the Wazuh indexer API from the command line.
Wazuh dashboard console: Navigate to the Wazuh dashboard upper-left menu ☰ > Indexer management > Dev Tools, paste the request, and click the play button. The console is already authenticated with your dashboard session.
Wazuh indexer API: Run the
curlcommands from any Wazuh indexer node or the Wazuh server. Replace<INDEXER_USERNAME>and<INDEXER_PASSWORD>with the Wazuh indexer credentials. ReplaceWAZUH_INDEXER_IPwith the IP address or FQDN of any Wazuh indexer node, and<REMOVED_WAZUH_INDEXER_IP>with the IP address of the node you are decommissioning.
Action |
Wazuh dashboard console |
Wazuh indexer API |
|---|---|---|
Verify the node and check the cluster name and version |
|
|
Check the cluster health |
|
|
List the cluster nodes and identify the elected cluster manager |
|
|
List the Wazuh indices |
|
|
Check how shards are allocated across the nodes |
|
|
View the cluster statistics |
|
|
View the persistent and transient cluster settings |
|
|
List the Wazuh data streams |
|
|
Verify the deployed Wazuh index templates |
|
|
List the ISM policies |
|
|
Check the ISM state of the Wazuh indices |
|
|
Verify that memory locking is enabled on every node |
|
|
Exclude a node from shard allocation before decommissioning it |
|
|
Clear the allocation exclusion after the node is removed |
|
|
Interpreting the outputs
Verifying the node with GET / returns the node name, the cluster name, and the version information:
Field |
Example value |
Description |
|---|---|---|
|
|
Name of the Wazuh indexer node that answered the request. |
|
|
Name of the Wazuh indexer cluster. |
|
|
Version of the Wazuh indexer engine. |
|
|
Apache Lucene version in use. |
Listing the nodes with GET _cat/nodes?v in a healthy three-node cluster produces an output similar to the following. The asterisk in the cluster_manager column marks the elected cluster manager node:
ip |
heap.percent |
ram.percent |
cpu |
load_1m |
node.role |
node.roles |
cluster_manager |
name |
|---|---|---|---|---|---|---|---|---|
10.0.0.10 |
34 |
68 |
12 |
0.42 |
dimr |
cluster_manager, data, ingest, remote_cluster_client |
indexer-1 |
|
10.0.0.11 |
29 |
64 |
9 |
0.31 |
dimr |
cluster_manager, data, ingest, remote_cluster_client |
indexer-2 |
|
10.0.0.12 |
31 |
66 |
10 |
0.36 |
dimr |
cluster_manager, data, ingest, remote_cluster_client |
indexer-3 |
Checking the cluster health with GET _cluster/health?pretty returns the overall state of the cluster:
Field |
Example value |
Description |
|---|---|---|
|
|
Name of the Wazuh indexer cluster. |
|
|
Overall health of the cluster. See the status table below. |
|
|
Total number of nodes in the cluster. |
|
|
Number of nodes that store data. |
|
|
Indicates that a cluster manager node has been elected. |
|
|
Number of active primary shards. |
|
|
Number of active primary and replica shards. |
|
|
Shards currently moving between nodes. |
|
|
Shards currently being created. |
|
|
Shards that could not be allocated to any node. |
|
|
Percentage of active shards. |
The status field takes one of the following values:
Status |
Meaning |
Recommended action |
|---|---|---|
green |
All primary and replica shards are allocated. |
None. The cluster is fully operational. |
yellow |
All primary shards are allocated, but one or more replica shards are unassigned. Data is available, but redundancy is reduced. |
Investigate in any deployment. Wazuh indices are created with |
red |
One or more primary shards are unassigned. Part of the data is unavailable. |
Check for stopped nodes, disk space, and allocation settings immediately. |
Replica shards
Wazuh creates its indices with index.auto_expand_replicas set to 0-1. The setting resolves to no replica on a single-node deployment, so that deployment reports green rather than yellow, and to one replica as soon as a second Wazuh indexer node joins. The Wazuh data streams that hold event and state data need no change.
Verify the replica configuration after you build a multi-node cluster, after you enable new detection content, and after an upgrade. Replace <INDEXER_USERNAME>, <INDEXER_PASSWORD>, and <WAZUH_INDEXER_IP> with the Wazuh indexer credentials and the IP address of any Wazuh indexer node.
Confirm that the Wazuh indices carry the automatic replica setting. Every Wazuh index should return
0-1.Wazuh dashboard console
Wazuh indexer API
GET wazuh-*/_settings?filter_path=**.auto_expand_replicas,**.number_of_replicas&expand_wildcards=allcurl -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> "https://<WAZUH_INDEXER_IP>:9200/wazuh-*/_settings?filter_path=**.auto_expand_replicas,**.number_of_replicas&expand_wildcards=all"The command output looks similar to this:
{ ".ds-wazuh-events-v5-system-activity-000001": { "settings": { "index": { "auto_expand_replicas": "0-1" } } }, ".ds-wazuh-active-responses-000001": { "settings": { "index": { "auto_expand_replicas": "0-1" } } }, ... }An index that returns
number_of_replicaswithoutauto_expand_replicashas a fixed replica count. Apply the setting to it with the command in step 3.Confirm that no Wazuh index is left without a replica. On a cluster with two or more nodes, this command returns no rows.
Wazuh dashboard console
Wazuh indexer API
GET _cat/indices/wazuh-*?v&h=index,pri,rep,health&s=rep,index&expand_wildcards=allcurl -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> "https://<WAZUH_INDEXER_IP>:9200/_cat/indices/wazuh-*?v&h=index,pri,rep,health&s=rep,index&expand_wildcards=all" | awk '$3 == 0'The command output looks similar to this:
index pri rep health .ds-wazuh-events-v5-system-activity-000001 1 1 green .ds-wazuh-active-responses-000001 1 1 green .ds-wazuh-events-raw-v5-000001 1 1 green ...
The Wazuh dashboard console command lists every index sorted by replica count, so read the rows at the top of the output. Every row should show
1in therepcolumn. The Wazuh indexer API command filters the list down to the indices with a replica count of 0 and prints nothing when the configuration is correct. A Wazuh index with0in therepcolumn on a multi-node cluster has a fixed replica count. Apply the setting to it with the command in step 3.Note
Running the same command without the
wazuh-*filter also returns the internal indices of the OpenSearch plugins. Several of those are created with a single copy of their data by design, so they appear with0in therepcolumn on a healthy cluster. Do not change them.Apply the automatic replica setting to a Wazuh index that failed step 1 or step 2. Replace
<INDEX_NAME>with the name of that index and repeat for each one.Wazuh dashboard console
Wazuh indexer API
PUT <INDEX_NAME>/_settings { "index": { "auto_expand_replicas": "0-1" } }curl -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> -XPUT "https://<WAZUH_INDEXER_IP>:9200/<INDEX_NAME>/_settings" -H 'Content-Type: application/json' -d '{"index": {"auto_expand_replicas": "0-1"}}'The command output looks similar to this:
{"acknowledged":true}Note
Do not send one request that covers several indices with a wildcard. The Wazuh indexer applies a settings request to all of its targets or to none of them, so a single rejected index makes the whole request fail and nothing is changed.
curlexits with code 0 in that case, so confirm that each response contains"acknowledged": true.