Configuring Disk Allocation Thresholds in Elasticsearch and OpenSearch – Source: socprime.com

When running an Elasticsearch or OpenSearch cluster, efficient disk space management is essential for ensuring stability and performance. These platforms provide configurable settings to manage how shards are allocated based on available disk space. Here, we discuss three key settings related to disk allocation thresholds:

1. cluster.routing.allocation.disk.threshold_enabled
This setting enables or disables disk-based shard allocation. When set to true, Elasticsearch/OpenSearch will monitor disk usage and allocate shards accordingly, ensuring nodes don’t become overwhelmed. If false, the cluster will ignore disk space considerations, which can lead to instability if nodes run out of storage.
Default: true

2. cluster.routing.allocation.disk.watermark.low
The low watermark is a soft threshold that defines the minimum amount of free disk space a node should have before Elasticsearch/OpenSearch starts avoiding allocating new shards to that node.

• It can be set as a percentage (e.g., 85%) or an absolute value (e.g., 100gb).
• When the available disk space on a node falls below this threshold, no new shards will be allocated to it, although existing shards will remain.

Example:

cluster.routing.allocation.disk.watermark.low: "85%"

This setting would avoid shard allocation if a node has less than 15% free disk space.

3. cluster.routing.allocation.disk.watermark.high
The high watermark is a stricter threshold that determines when Elasticsearch/OpenSearch starts actively relocating shards away from a node.

• Like the low watermark, it can be specified as a percentage or absolute value.
• If a node’s available disk space drops below this level, the cluster will move shards from this node to others with more capacity.

Example:

cluster.routing.allocation.disk.watermark.high: "90%"

Here, if a node has less than 10% free disk space, the cluster will redistribute shards to other nodes.

Use Case and Best Practices

• Monitoring Disk Usage: These settings are critical for preventing nodes from running out of storage, which can lead to shard failures and cluster instability.
• Cluster Scaling: Set thresholds based on your infrastructure’s growth potential. For example, if nodes are expected to fill quickly, lower watermarks may help distribute the load proactively.
• Combination of Absolute and Percentage Values: Use percentage values for dynamic systems where disk sizes may vary and absolute values for fixed-size storage environments.

Configuration Example
To configure these settings, add them to your cluster’s elasticsearch.yml or opensearch.yml file:

cluster.routing.allocation.disk.threshold_enabled: true cluster.routing.allocation.disk.watermark.low: "85%" cluster.routing.allocation.disk.watermark.high: "90%"

After updating the configuration, restart the nodes for the changes to take effect.

Was this article helpful?

Like and share it with your peers.