How to Adjust an Elasticsearch Cluster's Disk Allocation Watermark

Introduction

The storage requirements for Elasticsearch documents often exceed its default allocation, resulting in an allocation error. The solution to this problem is to increase the space available to Elasticsearch. This tutorial shows how to adjust Elasticsearch cluster disk allocation watermark .

Cluster Block Allocation Error

• FORBIDDEN/12/index read-only / allow delete (api)
• You may encounter the above error when you try to add some documents to an index in your cluster.

• A 403 error in HTTP indicates that your browser has attempted to access a page or some other resource that the server has disallowed.
• Elasticsearch considers the disk space available on a node before allocating shards to that node.
• You’ll typically encounter this error when the cluster Elastisearch is running on reaches its high watermark threshold, indicating that it’s starting to run out of disk space.

Ensure that you make setting changes correctly when reproducing this error. The steps for this procedure are as follows: Fill the Elasticsearch data disk until it exceeds the high disk watermark with this command: allocate -l9G largefile. Verify the high disk watermark is exceeded by reviewing the ES log. Mark the indices as read-only if you haven’t already. Change the settings and verify that it fails since writes are now prohibited. Fix the high disk watermark condition using the command rm largefile. Check that the ES log acknowledges that the node has gone under the high disk watermark and it’s now available to write to. Change the settings, which should fail even though disk allocation is now under the high disk watermark.

‘GET’ Cluster Information

‘GET’ Cluster Health

‘GET’ the state of your cluster

Solutions for Fix the Cluster Block Allocation Error

Free some space on your cluster’s hard disk:

• Free some disk space on the cluster, usually by deleting data.
• Contact the company hosting your server. You may be be able to pay extra to increase the volume of your drive, depending on your server type.

Adjust the high and low watermarks:

The cluster.routing.allocation.disk.watermark.high setting on your cluster determines the point at which Elasticsearch relocates shards away from a node on that cluster. The default value for this setting is "90%", but you can adjust it as follows:

Note: You must set the value for High Watermark below the value of cluster.routing.allocation.disk.watermark.flood_stage amount. The default value for the flood stage watermark is “95%”`.

• You can adjust the low watermark to stop Elasticsearch from allocating any shards if disk space drops below a certain percentage. The default value for this setting is “85%”, which you can change as follows:

Disable the disk allocation threshold:

Another solution is to set the disk allocation threshold switch to false to prevent allocations from taking effect as follows: `bash curl -XPUT ‘http://localhost:9200/_cluster/settings’ -H ‘Content-Type: application/json’ -d ‘{ “transient” : { “cluster.routing.allocation.disk.threshold_enabled” : false } }’ ` * Review Elastic’s Disk-based Shard Allocation settings for more information on disk allocation in Elasticsearch.

Conclusion

This tutorial described several ways to adjust the Elasticsearch cluster disk allocation watermark, which may become necessary when this cluster begins running out of space. The most straightforward method is to increase the space available on the cluster either by deleting files or increase the disk volume. Other solutions include adjusting the watermarks and disabling the disk allocation threshold.