keys /metadata only in memory permanently (not swapped out) & data in disk
Folks ,
Please let me know your expert opinion !
This is my post to another group ,
Does any one know if couchbase would keep only the keys in the memory as opposed to the entire data (and let demand paging /swap decide what should be in the memory or not).
If we have only keys in the memory ,if the searches we are doing in the database would only give one record out and the search keys are random then having only keys in the memory would make optimal use of memory i,e could load it once and have it in RAM for ever and use the key to get to the data location .
I we have entire data in memory then OS demand paging would be a performance hit on the system .
Please let me know if couchbase works the above way i am looking for .
Thank in advance
best regards
Subra
And i got a response from someone :
Hello,
Couchbase Server ( http://www.couchbase.com ) is using this approach (all keys and metadata are in memory) and the data (JSON Document or generic content) are loaded in memory when needed (accessed).
Here some interesting reading about Couchbase Architecture:
- Couchbase Architecture : http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-architectur...
- Detailed Blog: http://horicky.blogspot.fr/2012/07/couchbase-architecture.html
regards
Tug
Could you please double confirm that the couchbase keeps keys &metadata only in memory/cache and the data/documents are loaded into memory only when they are accessed , this way we can optimize RAM usage .
Please let me know your expert opinion .
Thanks
Subra
Sorry for resurrecting the thread. But does that mean a couchbase node can hold more keys than they amount of memory available (since old keys are evicted)?
All meta data must be stored in memory in Couchbase 2.0 and in earlier Couchbase versions. This means that there is a hard limit on the number of items that can be held in any cluster. You can of course add nodes or increase the amount of memory allocated to your servers in order to increase this limit. I also want to note that we see this as a critical issue that we are addressing. Couchbase 2.1 will have fixes that reduce the size of the meta data needed to store each item. This will allow clusters to store more meta items then in the current versions, but there will still be some hard limit. In Couchbase 2.2 we will work to allow full eviction of meta data so that there is no limit on the amount of items that can be held in a cluster. Below are the meta data sizes per item.
Couchbase 1.8.x, 2.0.x:
Resident item: 64 bytes + size of key
Non-resident item: 64 bytes + size of key
Couchbase 2.1:
Resident item: 56 bytes + size of key
Non-Resident item: (item has expiration or is locking) 48 bytes + size of key, (no expiration or locking) 40 bytes + size of key
Couchbase 2.2:
Resident item: 56 bytes + size of key
Non-Resident item: 0 bytes
The response you got is correct.
Couchbase will store all key and meta data in memory and this data will never be removed. Let's say you have 8GB of memory allocated to your bucket and 100,000 keys stored in Couchbase. The meta data might consume 100MB which means the remaining memory allocated will hold the data for each corresponding key and all data will also be persisted on disk. If the data takes up more then the remaining memory space then some of the data will only be held on disk. If you do a get on an item that is not in memory then we will fetch it from disk and place it in memory. If there is no memory space available then we will make space by evicting some keys that have not been used recently.
Also, we do not use the OS to page keys in and out of memory so you should not have to worry about your data being swapped out by the OS. The reason is that we want to be able to provide users consistent latencies for requests so that users can satisfy certain SLA's. When you use the OS to page things in and out of memory it is not possible to provide consistent latencies.