How To Fix The Forbidden 12 Read-Only API Error In Elasticsearch
Introduction
When you execute an API request to make changes to a document or index on an Elasticsearch cluster you may sometimes receive a [FORBIDDEN/12/index read-only / allow delete (api)] error that will block the request. The forbidden 12 read only error Elasticsearch and the read delete only API error Elasticsearchx000D typically happens when Elasticsearch reads the disk as being low on storage space and will therefore put itself into “read-only” mode. As Elasticsearch’s determination is based, by default, on the percentage of free disk space, this can occur on large disks even if you still have plenty of gigabytes of free disk space available. This article will provide you with some tips to show you how to prevent this error from occurring.
NOTE: All Elasticsearch versions 6.0 and newer will require a Content-Type header option anytime a cURL request is performed with a content body.
Elasticsearch will return a "Content-Type header" error if you omit the -H 'Content-Type: <span>application/json' header option in a cURL request with a JSON body in the request:
Why Elasticsearch returns this error, and won’t allow PUT requests
If you get an index read-only / allow delete error, it may be because the free disk space on the hard drive the Elasticsearch cluster is running on is too low:
Some Solutions:
Free up disk space on the hard drives that the cluster’s nodes are running on.
Increase the cluster.routing.allocation.disk.watermark settings.
Clean up the indices and remove duplicates using a tool like Curator.
Disable the index.blocks.read_only_allow_delete setting.
You can use the below _cat and _cluster APIs to get more information about your cluster:
1 2 3 | # get cluster health: curl -XGET 'http://localhost:9200/_cat/health?v&pretty=true' curl -XGET 'http://localhost:9200/_cluster/health?wait_for_status=green&pretty=true' |
Check on the status of the Elasticsearch cluster and its indices
1 | curl -XGET 'http://localhost:9200/_cat/indices?pretty=true' |
How to Fix the FORBIDDEN Read-Only / Allow Delete Error for Elasticsearch API Requests
Delete files on the Elasticsearch server to free up disk space
Free up enough space so that API write requests will be re-enabled on your Elasticsearch index by deleting any unwanted files on the hard drive of the server running the Elasticsearch cluster.
How to delete specific Elasticsearch documents and indices
Move or delete an Elasticsearch index from a node by using this cURL request:
1 | curl -XDELETE _localhost:9200/some_index |
Here’s the cURL request to delete a document on an Elasticsearch index:
1 | curl -XDELETE localhost:9200/some_index/_doc/someDocumentID_1234?pretty=true |
Use a tool such as Elastic’s Curator to clean up indices
Elasticsearch has a Curator tool, written in Python, that can help you manage you indices and sort through data and duplicate indexes.
How to change the disk.watermark` thresholds for the Elasticsearch cluster
Elasticsearch automatically disables certain processes on the cluster when the disk allocation free-space “watermark” drops below a certain level. When this happens, Elasticsearch will set the indices on the cluster to read-only. You can adjust the cluster.routing.allocation.disk.watermark settings to re-enable writing to the Elasticsearch index:
1 2 3 4 5 6 7 8 9 10 | curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "10gb", "cluster.routing.allocation.disk.watermark.high": "7gb", "cluster.routing.allocation.disk.watermark.flood_stage": "5gb", "cluster.info.update.interval": "1m" } } ' |
NOTE: In order to avoid an illegal_argument exception, as shown below, make sure the disk.watermark.low setting is higher than the disk.watermark.high number:
1 2 3 4 5 | "error": { "root_cause": [{ "type": "illegal_argument_exception", "reason": "low disk watermark [6gb] less than high disk watermark [7gb]" } |
Elasticsearch will force the index.blocks.read_only_allow_delete state onto any index that has one or more nodes that exceed this flood stage threshold.
Refer to our complete article on this topic for more details.
Disable the “read_only_allow_delete” setting for Elasticsearch indexes
You can also try to force the index to be writable by disabling the index.blocks.read_only_allow_delete setting by changing its value to either false or null.
How to change the read_only for all of indexes on an Elasticsearch cluster
Use the following cURL request to disable read_only_allow_delete setting for all of the indices on the cluster:
1 2 3 4 5 | curl -X PUT "http://localhost:9200/_all/_settings?pretty" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } ' |
Change the read_only_allow_delete setting to false instead of null:
1 2 3 4 5 | curl -X PUT "http://localhost:9200/_all/_settings?pretty" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": false } ' |
How to change the read_only for all of indexes on an Elasticsearch cluster using the Kibana Console:
1 2 3 4 | PUT _all/_settings { "index.blocks.read_only_allow_delete": null } |
How to change the read_only for a single index on an Elasticsearch cluster
You can change the read_only setting for a specific index with the following command:
1 2 3 4 5 | curl -X PUT "localhost:9200/some_index/_settings?pretty" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": false } ' |
Change the read_only_allow_delete setting for just one index in Kibana:
This request puts the setting to null:
1 2 3 4 | PUT some_index/_settings { "index.blocks.read_only_allow_delete": null } |
This request puts the setting to false:
1 2 3 4 | PUT some_index/_settings { "index.blocks.read_only_allow_delete": false } |
Conclusion
In this tutorial you learned how to address the FORBIDDEN Read-Only / Allow Delete Error for Elasticsearch API Requests. Remember, there are several options to correct these errors and you can use these strategies to stop getting the FORBIDDEN/12/index read-only / allow delete error.
The forbidden 12 read only error Elasticsearch and the read delete only API error Elasticsearch can be addressed by:
• Deleting specific Elasticsearch documents and indices to free up disk space.
• Changing the read_only_allow_delete setting.
• Changing the read_only for all of indexes on an Elasticsearch cluster.
• Changing the read_only for a single index on an Elasticsearch cluster.
• Changing the read_only setting for a specific index on an Elasticsearch cluster.
• Changing the disk.watermark thresholds for the Elasticsearch cluster.
Remember that all Elasticsearch versions 6.0 and newer will require a Content-Type header option anytime a cURL request is made with a content body. Also, bear in mind that in order to avoid an illegal_argument exception the disk.watermark.low setting must be higher than the disk.watermark.high number.
Pilot the ObjectRocket Platform Free!
Try Fully-Managed CockroachDB, Elasticsearch, MongoDB, PostgreSQL (Beta) or Redis.
Get Started