Resizing Persistent Volume Claims (PVCs) for OpenSearch in Kubernetes: Challenges and Solutions – Source: socprime.com

Managing storage requirements for an OpenSearch cluster deployed via a Helm chart in Kubernetes can present unique challenges, especially when scaling Persistent Volume Claims (PVCs) for StatefulSets. PVC resizing in Kubernetes is a straightforward concept, but its implementation may run into issues depending on the underlying storage class, StatefulSet behavior, and OpenSearch’s requirements. This article delves into the common problems encountered during PVC resizing and presents a practical solution.

Problem: PVC Resizing in StatefulSets

StatefulSets are commonly used for deploying stateful applications like OpenSearch, which require consistent storage to maintain data integrity. When resizing PVCs associated with StatefulSets, several issues can arise:

1. Storage Class Constraints: Not all storage classes support dynamic PVC resizing. For example, some CSI drivers require the volume to be unmounted before resizing.
2. Helm Chart and StatefulSet Behavior: By design, StatefulSets don’t allow direct PVC modifications. Helm charts managing OpenSearch may not account for PVC changes dynamically, making resizing a manual process.
3. Downtime Considerations: Since PVC resizing might involve pod restarts or volume unmounting, careful planning is required to minimize downtime.
4. Manual Intervention: For OpenSearch clusters with multiple nodes, resizing PVCs across all nodes requires repetitive manual operations if not automated.

Solution: Patching PVCs in Kubernetes

To address the challenges of resizing PVCs for OpenSearch, you can use Kubernetes’ kubectl patch command to update the storage size directly. Below is a step-by-step guide to resizing PVCs for an OpenSearch deployment.

Prerequisites

1. Verify Storage Class Support: Ensure your storage class supports PVC resizing. For dynamic resizing, the allowVolumeExpansion field in the storage class must be set to true.
2. Enable Resize Feature: Make sure Kubernetes has the feature gate for PVC resizing enabled (most modern versions have this as a default setting).
3. Backup Data: Always back up OpenSearch data before making storage modifications to prevent potential data loss.

Resizing Data PVCs

In an OpenSearch cluster, the data nodes typically hold the bulk of the stored information. To resize the PVCs associated with these nodes:

kubectl patch -n opensearch pvc data-0 -p '{ "spec": { "resources": { "requests": { "storage": "256Gi" }}}}' kubectl patch -n opensearch pvc data-1 -p '{ "spec": { "resources": { "requests": { "storage": "256Gi" }}}}' kubectl patch -n opensearch pvc data-2 -p '{ "spec": { "resources": { "requests": { "storage": "256Gi" }}}}'

Resizing Master PVCs

The master nodes typically require less storage compared to data nodes. Resize the PVCs for master nodes as follows:

kubectl patch -n opensearch pvc master-0 -p '{ "spec": { "resources": { "requests": { "storage": "128Gi" }}}}' kubectl patch -n opensearch pvc master-1 -p '{ "spec": { "resources": { "requests": { "storage": "128Gi" }}}}' kubectl patch -n opensearch pvc master-2 -p '{ "spec": { "resources": { "requests": { "storage": "128Gi" }}}}'

Additional Steps

• Monitor Pods: After resizing the PVCs, the StatefulSet pods may restart to apply the changes. Use the following command to monitor pod statuses:

kubectl get pods -n opensearch

• Verify Storage Size: Confirm the updated storage size using:

kubectl describe pvc  -n opensearch

Conclusion

Resizing PVCs for an OpenSearch deployment in Kubernetes requires careful consideration of storage class capabilities, StatefulSet behavior, and potential downtime. By leveraging the kubectl patch command, administrators can dynamically resize PVCs and maintain the stability of their OpenSearch clusters. With proper planning and automation, this process can be streamlined to ensure seamless scalability.