Periodically I see a sudden "dent" in the number of items on the DB, matched by a drop in memory stats (even the watermarks...?) and a surge in the write queue.
Funny thing is the items count comes back in a few seconds time.
Any ideas what that is?
How often this thing is happening or you cluster?
Can you tell us more about your environment:
- Size of the cluster
- which client SDK is used to connect to your server?
And you application does not happen to delete and recreate documents? (for example delete with a TTL, and some injector...)
let me try to cover your questions.
This is happening every few minutes. Actually, as I am keeping an eye on the monitor right now, I saw it happen three times in less than 10 minutes time, one of them being a double one.
Unfortunately, if I zoom out the monitor, I lose the details and the inverted spikes do not show. If there is a way to get this information from some logs, I would love to know it.
Ubuntu server 12.04.1 64bits
4 Nodes x 16GB RAM (14.7 for Couchbase) - 43GB HardDisk space
c# Assembly Couchbase.dll, v2.0.50727
About the application now, in a nutshell, I store documents with an validFor timespan as in:
StoreMode.Set, id.ToString(), memory.GetBuffer(), Properties.Settings.Default.StateExpiration);
However, I do not store in batches, but rather at a more-or-less steady per-item flow, plus I have no recreate functionality.
Another note: It appears that the massive write queue is not a consistent "symptom", as you can see in the following screenshot, where the entire write queue on the chart is the single digits.
The item count reduction remains steady at 100k items
It happens because one of the nodes in your cluster didn't return data for that sample. If you click on the arrow at the bottom of the "little" graph with the stats that look funky and choose "show by server" you'll notice one (or possibly more) servers didn't supply data at that time.
right you are, just checked it live.
this is expected (or at least tolerated) I presume?
Not sure. I don't work for CB. Me just pawn in game of life.
you work for mongo then? :)
So Pgrigor is correct this happens when some of the stats requests are not returned soon enought so the console does not show them.
This has not impact on the data itself and the performance of the application that consumes the data.
thank you for the input tgrall.
© 2013 COUCHBASE All rights reserved.