Couchbase
  • Why NoSQL?
  • Couchbase Server
  • Download
  • Resources
  • Careers
Home | Forums | Membase | Membase Server 1.7.x

how many records can be holden in membase server ?

6 replies [Last post]
  • Login or register to post comments
Wed, 08/10/2011 - 00:19
kuerant
Offline
Joined: 08/09/2011
Groups: None

Hi, all,

I am testing membase 1.7.1.
I setup a membase server with 1GB RAM + 1TB disk, with a 'default' bucket, bucket-type is membase.

I insert record into membase, using libmemcached library, in binary-protocol. for each record, key is 10 Bytes, value is about 50 Bytes. After insert about 7M records, the libmemcached API got MEMCACHED_PROTOCOL_ERROR, and always the same error, even I tried to restart client.

On web console, it shows that less than 100MB memory is UNUSED, and only 500 MB disk is USED. and on "VBUCKET RESOURCES" figure, it shows "6.97M items" with a line drawn in the top of picture. I thought the membase is FULL!

Is it true that "the bucket is FULL" ?? I want to save more data, and I can deal with a little more latency, but the membase seemed to refuse that? Why ? and How ??

Is there any other log for me? I just know display log using mbbrowse_log, it shows no error, but no more data could be inserted into membase.

please help !

- ant

Top
  • Login or register to post comments
Thu, 08/11/2011 - 17:01
mikew
Offline
Joined: 03/14/2011
Groups:

There is technically a limit on the amount of data you can store in Membase. This limit is created by the amount of key meta data that can be held in memory. This limit can be raised by adding more memory. With that said, I don't think you are hitting that limit. I think what is happening is that the server is sending you temporary out of memory errors and you client isn't correctly dealing with these issues. If you get them it means that your client is sending data to the server faster than the data can be persisted. The correct behavior is to backoff for a few milliseconds and then start sending the server more data. See if you can figure out exactly what error is coming back from the server, MEMCACHED_PROTOCOL_ERROR is too general.

Top
  • Login or register to post comments
Thu, 08/11/2011 - 20:15
kuerant
Offline
Joined: 08/09/2011
Groups: None

Thanks mikew.

I use libmemcached in C to talk to membase.
my codes :

memcached_return_t rc = memcached_set( _mc, key.data(), key.size(), value.data(), value.size(), expiration, flags );
if ( MEMCACHED_SUCCESS != rc ) {
THROW_FMT( "fail to set key '%s' in memcache -- [%d] %s", key.c_str(), rc, me mcached_strerror(_mc, rc) );
}

the execption says it is a MEMCACHED_PROTOCOL_ERROR, and I donot know how to get more detail error message.

Thanks for any suggestion.

Thanks!

- ant

Top
  • Login or register to post comments
Thu, 08/11/2011 - 20:46
kuerant
Offline
Joined: 08/09/2011
Groups: None

There is technically a limit on the amount of data you can store in Membase. This limit is created by the amount of key meta data that can be held in memory.
^^^^^^^^^^^^^^^^^
Does this mean that all keys ( or metadata of keys ) must be resident in memory? If the memory is full ( quota is used up ), no more keys would be insert into ??

Is there a document describing this ?

Thanks.

Top
  • Login or register to post comments
Mon, 08/15/2011 - 15:20
perry
Offline
Joined: 10/11/2010
Groups:

Yes, you are correct that all keys and metadata need to be stored in RAM and that you can fill up your memory with this data if not monitored appropriately.

Here's a document describing how the system reacts when memory gets full: http://www.couchbase.org/wiki/display/membase/Growing+Data+Sets+Beyond+M...

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!

Top
  • Login or register to post comments
Wed, 08/17/2011 - 22:38
kuerant
Offline
Joined: 08/09/2011
Groups: None

Thanks, Perry.

But that documents does NOT mention that "all keys and metadata need be stored in RAM". Is there another document?

thanks.

Top
  • Login or register to post comments
Tue, 08/23/2011 - 16:34
perry
Offline
Joined: 10/11/2010
Groups:

Not explicitly, what kind of information were you looking for? You are correct that all keys and metadata need to be stored in RAM. You might want to look at the sizing guidelines as well: http://www.couchbase.org/wiki/display/membase/Sizing+Guidelines

Perry

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!

Top
  • Login or register to post comments
  • Login or register to post comments
  • Login
  • Register

Company

  • About Us
  • Leadership
  • Customers
  • Partners
  • Contact Us

Product

  • Couchbase Server
  • Couchbase SDKs
  • Use Cases
  • Documentation
  • Forums

Open Source

  • Couchbase Project
  • Couchbase vs. CouchDB

Commercial

  • Subscriptions & Support
  • Training & Services

News

  • Blog
  • Newsletter
  • Press Releases
  • Buzz

Follow Us

    
  • Customer Login
  • Terms of Service
  • Privacy Policy
  • Trademark Policy
  • Site Map

© 2013 COUCHBASE All rights reserved.

Sign in to Couchbase Community

close
  • Create new account
  • Request new password
You are logging into the Forums, Wiki and Issue Tracker