Substential decrease in recent insert operation on a single bucket

Couchbase Server
1

Hello all,

I’ve been working with Couchbase community edition server for quite a while now. My main purpose is storing log records from various systems. Until recently, the insert rate was about ~2000 ops/sec.
I now have a single bucket full of 80,000,000 documents, and I’ve seen a significant drop in the rate of insert all the way down to ~40 ops/sec.
I’ve added a new server instance to act as a new node in my cluster (my cluster now has 2 nodes), rabalanced them. Furthermore, I’ve changed the max_parallelism value of each server to 4 (the maximum allowed in community edition). I’ve tried setting the bucket’s ejection policy to both “value only” and “full”.
All of the described steps didn’t make any significant change. My current rate of inserts is ~80 ops/sec.
I’m a bit confused on what could have made this substantial rate drop happen, and would like to know if there is a way to diagnose and even fix the issue.

Thanks in advance.

2

@0rka,

I don’t think this bought you much except gaining a replica vs before with one node you had no replica.

Are you seeing TEMP_OOM errors?

cb-admin-gui-stats
cb-admin-gui-stats764×657 41 KB

What % of your data is in memory (active resident %)?

23%20AM
23%20AM876×302 23.4 KB

Are you in AWS?
Did you use up all you IOPS credits to disk and AWS is throttling you?

3

Hey @househippo, thanks for the response!

First of all I haven’t seen any TEMP_OOM errors ever, so I guess this is fine.
Second, My average Active Docs Resident percentage is at about 60%-65%.

I’m not running my servers in AWS instances. All of my servers are on my local physical ESXi instances.

4

@0rka,

If you are not getting temp_oom Couchbase Server is able to do writes.

Could you check your timings for operations via Couchbase CLI Command cbstats.

To get a more accurate view on performance on GET/SET …etc and any disk operations you can use
cbstats timings. Hope fully it should all be in microseconds and some milliseconds.

#opt/couchbase/bin/cbstats localhost:11210 timings -b {bucket_name} -p {bucket_password}

https://developer.couchbase.com/documentation/server/current/cli/cbstats/cbstats-timing.html

if you put in localhost:11210 you will get stats of the localhost.
if you put in localhost:11211 it will get Moxie to pull stats from the whole cluster.

get_cmd = GET()s
store_cmd or cmd_store = SET()s.