Is it possible to suggest eviction?
Hello,
I'm planning to use a couchbase bucket for both data storage and database.
This means that I have essentially 2 kind of documents stored in the bucket:
1. JSON like objects which will be accessed through views (~ few Kb data)
2. Binary data, just key indexed (~ few hundreds Mb, 100-300Mb), but on which I need MVCC for detecting concurrency issues.
I understand that attachments are not supported yet, and I don't care, I can very well assign a key to my BLOBs and store them as independent documents.
What I find "funny" is the 20Mb limit on BLOBs, but for that I can use a client workaround and split it in chunks.
What I really can't avoid is the Temporary Failure error when I'm uploading on the Couchbase too much data at once.
I understand that I can "retry later", because the eviction system is evicting (flushing on disk) some of my documents, but I don't know how much later I will have to try.
Since I know that my BLOBs don't need sub-millisec access, can I suggest to the Couchbase to evict them as soon as possible? This way I wouldn't have to cope with Temporary Failure that often.
If you think I'm missing something and you can suggest me a better way to achieve the purpose, please suggest.
Thank you,
Federico
EDIT:
The point I'm trying to make is: there are some documents for which I need low latency, but for others I don't. For large BLOBs I typically don't need low latency (network data transfer is going to dump the performance anyway), therefore I'd like those BLOBs not to stay in RAM on my Couchbase
Firstly I'd like to thank you for your answer.
If the 90% limit for Temporary Failure is hardcoded or somehow readable from within the client I could try to write only when the bucket memory usage is below that.
I'll look into the docs and try to understand how to read the bucket memory consumption.
Secondly I'm using the python client library, but I plan to use the Java and C clients as well.
I tried the evict command, but I've probably done something wrong, because I couldn't see behavior differences.
The code I tried with the evict() is somehow like this small extract
cb = CbClient("http://localhost:8091/pools/default","default","",False) while( not data.isEmpty() ): chunkKey = generateChunkKey() chunk = consumeData(data) cb.set( chunkKey, 0, 0, chunk) memcached = cb.memcached(chunkKey) memcached.evict_key(chunkKey)
Am I doing the eviction right?
Your code looks correct. In order to make sure you are doing things correctly you can check the server stats. Here's how:
Do a set from your python client.
/opt/couchbase/bin/cbstats : all | grep vb_active
You will see a stat that says "vb_active_num_non_resident". This is the amount of active items that are not in memory on the server you connected to. Remember this number. Also, for testing make sure that your "vb_active_perc_mem_resident" stat is 100. The reason is that if it is not then the key you choose to evict might have already been evicted. (It's not bad if this isn't 100, but it might lead to this test failing even though everything is working properly)
Now evict the key you just set.
/opt/couchbase/bin/cbstats : all | grep vb_active
You should see "vb_active_num_non_resident" be one more than the last time you ran the command.
Let me know if you have any problems with this.
I have a problem indeed: everything works perfectly as you said if I input the eviction command in Python interactive mode, leading to this output on the Python console
>>> cb = CbClient("http://localhost:8091/pools/default","default","",False) >>> fakeDoc = {"type": "fake"} >>> fakeDocKey = "dummy" >>> fakeDocStatus = cb.set(fakeDocKey, 0, 0, json.dumps(fakeDoc)) >>> memcached = cb.memcached(fakeDocKey) >>> memcached.evict_key(fakeDocKey) (3979266754, 0, 'Ejected.')
and increasing the "vb_active_num_non_resident" by one.
When instead I run a non interactive code doing the same thing (1 set and 1 eviction) I get almost random behavior:
usually the "vb_active_num_non_resident" is unchanged, sometimes it gets changed in a very large number (seems like an overflow or something).
If i run a whole loop with some hundreds of insertions and evictions I get the same result as with just 1 set and 1 eviction.
FYI I'm running couchbase on a single machine, OS: x86_64-unknown-linux-gnu Version: 2.0.0 community edition (build-722-rel) on Ubuntu 12.04.
PS. While doing these tests I repeatedly flushed and/or deleted and recreated the bucket, and I used this command:
watch -d "/opt/couchbase/bin/cbstats localhost:11210 all | grep vb_active"
to monitor the parameters.
I believe that the tests led the couchbase in an inconsistent state, because now it's repeatedly restarting, with this message on the web-UI: "Difficulties communicating with the cluster; displaying cached information!".
Restarting the linux service doesn't help, I think I'll reinstall it.
Thanks for reporting this. I want to mention that the underflow issue has been fixed for Couchbase 1.8.1 which should be released in the next couple of weeks.
http://www.couchbase.com/issues/browse/MB-5418
I am also going to pass this issue onto our QE team. You can follow this issue here:
http://www.couchbase.com/issues/browse/MB-5596
Thanks for reporting this issue.
No short answer is that Couchbase is already doing this for you given you workload. Let's say you have a new cluster with no data in it and you start loading in items faster than Couchbase can write to disk. WHen you hit 75% capacity Couchbase base will begin ejecting items and at 90% capacity you start getting Temporary Failure messages. So basically when you being getting close to maximum capacity Couchbase will have already started the ejection process. The reason you are still getting Temporary failures is because Couchbase must keep items in memory until they have been persisted. As a result the temporary failure means that all of the memory in Couchbase is in a state that it cannot be evicted. Once some items are persisted Couchbase will eject them and the eject their values from memory.
What I would recommend doing is having your application try to figure out what the best loading rate (ops/sec) is. In the past I have used algoriths similar to TCP congestion control algorithms and they have greatly helped me avoid hitting temporary failure errors.
On another note we have an evict command, but it is not implemented in all of our libraries. If you tell me which client you are using I can add it as a feature for the next release if it doesn't exist.