CB 8.x New Feature w.r.t Filesystem Usage Limits

Couchbase Server
1

Hi Team,

In Couchbase 8.x, a new feature allows configuring a file system threshold value. When this threshold is reached, the data service stops writing to storage paths. We want to know if application code requires changes, such as exception handling, to detect when the threshold is reached.

Thanks,

Debasis

2

Hi Debasis,

The Couchbase SDKs report an error if you try something that would exceed the limit.

The error might look different depending on which SDK language and version you use. As a data point, Java SDK 3.10.0 throws a generic CouchbaseException.

We’re currently exploring whether it makes sense for the SDKs to return a more specific exception in this case.

Do you have feedback on whether your application would handle a hypothetical “filesystem limit exceeded” exception differently from any other kind of storage failure?

Thanks,
David

1 Like
3

Here is an example of the errors I got when I enabled the 8.0 Data Service disk usage limit and ran into the limit.

Example – Java client using Couchbase java-client version 3.8.2:

com.couchbase.client.core.error.CouchbaseException: UpsertRequest failed with unexpected status code UNKNOWN

{
“completed”: true,
“coreId”: “0x65a11a7a00000001”,
“idempotent”: false,
“lastChannelId”: “65A11A7A00000001/0000000078778567”,
“lastDispatchedFrom”: “172.168.0.201:51868”,
“lastDispatchedTo”: “abc-abcd-node-001.oaq8p0ij-sbtblbe.couchbase.com:11207”,
“requestId”: 9,
“requestType”: “UpsertRequest”,
“retried”: 0,
“service”: {
“bucket”: “bucket4”,
“collection”: “_default”,
“documentId”: “000001”,
“errorCode”: {
“description”: “Disk space where the bucket is deployed is above the configured threshold, writes to this bucket is rejected”,
“name”: “BUCKET_DISK_SPACE_TOO_LOW”
},
“opaque”: “0xe”,
“scope”: “_default”,
“type”: “kv”,
“vbucket”: 65
},
“status”: “UNKNOWN”,
“timeoutMs”: 2500,
“timings”: {
“dispatchMicros”: 83155,
“encodingMicros”: 1137,
“totalDispatchMicros”: 83155,
“totalServerMicros”: 0,
“totalMicros”: 102030,
“serverMicros”: 0
}

Example - from Query Workbench

INSERT INTO `bucket4`._default._default(KEY, VALUE) VALUES (“mykey3”, ‘{“field1”:“note”}’);

Result:

[
{
“code”: 12009,
“msg”: “DML Error, possible causes include concurrent modification. Failed to perform INSERT on key mykey3 - cause: Error in INSERT of key: mykey3”,
“reason”: {
“_level”: “error”,
“caller”: “couchbase:2851”,
“cause”: {
“cause”: {
“_level”: “exception”,
“caller”: “memcached:580”,
“cause”: {
“attempts”: 1,
“cause”: “BUCKET_DISK_SPACE_TOO_LOW”
},
“code”: 5502,
“key”: “datastore.couchbase.bucket.action”,
“message”: “Unable to complete action after 1 attempts”
},
“key”: “mykey3”
},
“code”: 12036,
“key”: “datastore.couchbase.insert.error”,
“message”: “Error in INSERT of key: mykey3”
}
}
]

Example - From python (KV insert)

% python3 ./kv_insert.py

<ec=1004, category=couchbase.network, message=protocol_error (1004), context=KeyValueErrorContext:{‘retry_attempts’: 0, ‘key’: ‘document-key’, ‘bucket_name’: ‘bucket4’, ‘scope_name’: ‘_default’, ‘collection_name’: ‘_default’, ‘opaque’: 7, ‘status_code’: 56, ‘error_map_info’: {‘code’: 56, ‘name’: ‘BUCKET_DISK_SPACE_TOO_LOW’, ‘description’: ‘Disk space where the bucket is deployed is above the configured threshold, writes to this bucket is rejected’, ‘attributes’: {1, 18, 19}}}, C Source=/Users/couchbase/jenkins/workspace/python/sdk/python-scripted-build-pipeline/py-client/src/kv_ops.cxx:680>

Also, if you enable the Data disk usage limit, you can monitor whether the limit has been reached using the Prometheus stat/metric “cm_resource_limit_reached” with resource=”kv_disk_usage”.

Example:

cm_resource_limit_reached{instance="172.31.43.209:8091", job="test-cluster1", resource="kv_disk_usage", severity="maximum"} value will be 1 if the limit is reached

1 Like