Couchbase deleting old documents automatically

Couchbase Server
1

Hi Team,
We are using Couchbase for 1 POC where we are storing around 4-5 lakh documents/day. But, We started saving records from Oct 2020 and continued till now. But, As of now, We can’t see any records from Oct month and no.of records also reduced automatically.
While working on POC, We added 3 servers to the Couchbase servers and initially it was working fine and showing expected no.of documents on each server.

We are using the Enterprise version [6.6.0] of Couchbase on Centos 7 machine.

2

You might have set expiration at document level or bucket level check it out. https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/expiration.html

3

Bucket Max Time-To-Liv is not enabled at bucket level.

4

Hi Abhijeet,
There could be multiple reasons for this loss:

  1. TTL could be set at bucket level or at document level. Since you have mentioned there was no TTL setup at bucket level, it could be that TTL was set at the document level while inserting those documents. Could you please check

  2. What is the bucket type. If its Ephemeral bucket, was its RAM size reduced? please check the ejection policy for Ephemeral buckets here: https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/buckets.html

  3. If not please let us know how that data was inserted eg: which SDK and which version

5

1.TTL could be set at bucket level or at document level. Since you have mentioned there was no TTL setup at bucket level, it could be that TTL was set at the document level while inserting those documents. Could you please check : No, we havent specified expiration while inserting documents.

  1. Bucket type is Couchbase : Is there ejection policy for couchbase bucket type as well ? We did create a cluster of 4 Servers but never reduced RAM capacity.

3.Data was inserted from freeradius : Here is the complete stack :
couchbase-release.x86_64 1.0-4 @/cb-temp
couchbase-server.x86_64 6.6.0-7909 @couchbase-server
libcouchbase-devel.x86_64 2.10.6-1.el7 @couchbase
libcouchbase2-bin.x86_64 2.10.6-1.el7 @couchbase
libcouchbase2-core.x86_64 2.10.6-1.el7 @couchbase
libcouchbase2-libev.x86_64 2.10.6-1.el7 @couchbase

6

Hi @abhijeet.kulkarni,
1.Does this bucket have any replica’s.
2. Were there any hard failovers on this cluster
Please check the Hard Failover Data Loss Warning in the below document

https://docs.couchbase.com/server/current/manage/manage-nodes/failover-hard.html#:~:text=The%20dialog%20provides%20the%20following,failure%20of%20in-flight%20operations.

  1. Are you observing that the lost documents are only from Oct ??
    If the data loss is from a specific time period then the reason could be due to TTL.
    If the data loss is random, then I suspect its due to a bad Failover