Ejection slow down under constant high set
I am wondering if I am missing something in the following problem:
My program (using the java client) inserts a lot of items to CB when it's started up. The set rate (e.g. 500 items/s) should be constant and sustainable (i.e. disk queue size should goes up and down to zero).
This works fine until we hit the high water mark. At this point, CB should eject items from the memory as they are all persisted. This happened as planned. However, the ejection rate decreased overtime. So the first time, the ejection is about 20k, the next time it is about 18k ... This happened until the ejection rate got lower than the set rate (500 items/s). At this point the temp OO occurred and the set operations failed. This happened until a point when every set operation failed. While all of this happened the disk queue got emptied regularly. When every set requests fail, the disk queue is zero.
Then after a few minutes, the eject started again. But the same thing happened. The ejection rate decreased and decreased until all the set failed. This cycle started again and again.
Did someone encounter this before? I can't find a bug report that describes this problem. Is there are tuning option I should change?
Thank you
I am running 1.8.0 on Linux. I also tested it on a 1.8.0 on Windows as well.
There is no temp OO until the ejection rate got slower than the set rate.
How do I do back off with the Java client? I am getting the state from future.getStatus().isSuccess(). I can see the status from future.getStatus().getMessage() but that is a string. Is there a status code returned somewhere?
Thanks
I wouldn't expect the ejection rate to slow down. Which version of the server are you running?
Also, are you doing a backoff when you hit temp OOM? You should be able to backoff with the Java client and then keep going.