OpenSearch: Cluster Blocks Read-Only – Source: socprime.com

OpenSearch can enforce read-only states on clusters or indices to protect against issues like low disk space or cluster instability. Understanding and resolving these blocks is crucial for maintaining a healthy and operational cluster. Below is a guide to address common scenarios.

How to Resolve cluster.blocks.read_only

The cluster.blocks.read_only setting typically occurs when OpenSearch detects a critical issue, such as running out of disk space. The cluster automatically sets itself to read-only to prevent further damage.

Steps to resolve:

1. Free up disk space:
2. • Delete unnecessary indices.
   • Archive old data.
   • Increase disk capacity.
3. Clear the read-only block: Run the following command to remove the block after resolving the disk space issue:

PUT _cluster/settings {   "persistent": {     "cluster.blocks.read_only": false   } }

Index-Level Read-Only Block

At the index level, OpenSearch can set specific indices to read-only, preventing writes. This is often triggered by disk watermarks or manual configuration.

Steps to resolve:

1. Identify the affected indices using:

GET _cat/indices?v

  2.  Clear the block for an individual index:

PUT //_settings {   "index.blocks.read_only": false }

  3.  If multiple indices are affected, use a wildcard:

PUT /*/_settings {   "index.blocks.read_only": false }

How to Resolve cluster.blocks.read_only_allow_delete

This block allows only delete operations on indices, typically to help free up space when disk watermarks are exceeded.
Steps to resolve:

1. Free up disk space as described earlier.
2. Clear the read_only_allow_delete block with:

PUT _cluster/settings {   "persistent": {     "cluster.blocks.read_only_allow_delete": false   } } 

Index-Level Read-Only Delete Block

When indices are individually set to read_only_allow_delete, write operations are blocked except for deletions.
Steps to resolve:

1. Check index settings:

GET //_settings

  2.  Remove the block for a specific index:

PUT //_settings {   "index.blocks.read_only_allow_delete": false }

  3.  To remove the block from all indices:

PUT _all/_settings {   "index.blocks.read_only_allow_delete": false }

Conclusion