how many records can be holden in membase server ?
Hi, all,
I am testing membase 1.7.1.
I setup a membase server with 1GB RAM + 1TB disk, with a 'default' bucket, bucket-type is membase.
I insert record into membase, using libmemcached library, in binary-protocol. for each record, key is 10 Bytes, value is about 50 Bytes. After insert about 7M records, the libmemcached API got MEMCACHED_PROTOCOL_ERROR, and always the same error, even I tried to restart client.
On web console, it shows that less than 100MB memory is UNUSED, and only 500 MB disk is USED. and on "VBUCKET RESOURCES" figure, it shows "6.97M items" with a line drawn in the top of picture. I thought the membase is FULL!
Is it true that "the bucket is FULL" ?? I want to save more data, and I can deal with a little more latency, but the membase seemed to refuse that? Why ? and How ??
Is there any other log for me? I just know display log using mbbrowse_log, it shows no error, but no more data could be inserted into membase.
please help !
- ant
Thanks mikew.
I use libmemcached in C to talk to membase.
my codes :
memcached_return_t rc = memcached_set( _mc, key.data(), key.size(), value.data(), value.size(), expiration, flags );
if ( MEMCACHED_SUCCESS != rc ) {
THROW_FMT( "fail to set key '%s' in memcache -- [%d] %s", key.c_str(), rc, me mcached_strerror(_mc, rc) );
}
the execption says it is a MEMCACHED_PROTOCOL_ERROR, and I donot know how to get more detail error message.
Thanks for any suggestion.
Thanks!
- ant
There is technically a limit on the amount of data you can store in Membase. This limit is created by the amount of key meta data that can be held in memory.
^^^^^^^^^^^^^^^^^
Does this mean that all keys ( or metadata of keys ) must be resident in memory? If the memory is full ( quota is used up ), no more keys would be insert into ??
Is there a document describing this ?
Thanks.
Yes, you are correct that all keys and metadata need to be stored in RAM and that you can fill up your memory with this data if not monitored appropriately.
Here's a document describing how the system reacts when memory gets full: http://www.couchbase.org/wiki/display/membase/Growing+Data+Sets+Beyond+M...
Thanks, Perry.
But that documents does NOT mention that "all keys and metadata need be stored in RAM". Is there another document?
thanks.
Not explicitly, what kind of information were you looking for? You are correct that all keys and metadata need to be stored in RAM. You might want to look at the sizing guidelines as well: http://www.couchbase.org/wiki/display/membase/Sizing+Guidelines
Perry
There is technically a limit on the amount of data you can store in Membase. This limit is created by the amount of key meta data that can be held in memory. This limit can be raised by adding more memory. With that said, I don't think you are hitting that limit. I think what is happening is that the server is sending you temporary out of memory errors and you client isn't correctly dealing with these issues. If you get them it means that your client is sending data to the server faster than the data can be persisted. The correct behavior is to backoff for a few milliseconds and then start sending the server more data. See if you can figure out exactly what error is coming back from the server, MEMCACHED_PROTOCOL_ERROR is too general.