Python & Memcached Bucket - Does it work?

Jon-Eric,

Sorry for your trouble. Could you post some minimal Python code (like something you'd try in IDLE) that reproduces the bug? I need to see which classes you're using mostly, but getting as much of the surrounding context as possible will be helpful.

Thanks,
Benjamin

Benjamin,

I'll be happy to post what I've tried. But, I've tried A LOT of different approaches.

It might be faster if you could just show me one way to successfully fetch a document from a Memcached bucket on a v1.8.1 server.

In the mean time, I'll try and put together a collection of what I've tried.

Thanks!

-Jon-Eric

Sure, the 'tests' directory has several examples:
https://github.com/couchbase/couchbase-python-client/tree/master/couchba...

test_client.py tests the "unified" client class Couchbase() (previously Server())
test_memcachedclient.py tests a direct memcached connection (generally at 11211 via moxi)
test_vbucketawareclient.py extends the MemcachedClient with some Couchbase specific features (gat, touch)
test_couchbaseclient.py is a dispatching client, but it doesn't have it's own tests as it's mainly used via Couchbase()

If you want to actually run those tests (vs. using them for reference) be sure it's on a *different* cluster than your development or production cluster as it will delete data and throw all kinds of junk into the 'default' bucket--it's only meant for SDK development and *not* for dev/production cluster testing.

Later.

Benjamin,

Here's what I've tried so far to retrieve a document from a (non-default) memcached bucket:

import couchbase
import couchbase.couchbaseclient
 
HOST = "couchbasevm_01"
PORT = 8091
USER = "Administrator"
PASSWORD = ""
COUCHBASE_BUCKET = "dev_cb"
MEMCACHED_BUCKET = "dev_mc"
COUCHBASE_DOCUMENT = "SomeCouchbaseDocument"
MEMCACHED_DOCUMENT = "SomeMemcachedDocument"
 
###############
# Version 0.6.0
###############
client = couchbase.Server("{}:{}".format(HOST, PORT), USER, PASSWORD)
bucket = client.bucket(MEMCACHED_BUCKET) # Fails with BucketUnavailableException.
bucket = client.bucket(COUCHBASE_BUCKET) # Fails with BucketUnavailableException.
 
client = couchbase.couchbaseclient.MemcachedClient(HOST, PORT)
client.version()    # Fails with EOFError: Got empty data (remote died?). from couchbasevm_01
 
# This eventually fails with Exception: vbucket map is not ready for bucket dev_mc
client = couchbase.couchbaseclient.VBucketAwareCouchbaseClient("http://{}:{}/pools/default".format(HOST, PORT), MEMCACHED_BUCKET)
 
# Fetching a document from a couchbase bucket succeeds.
client = couchbase.couchbaseclient.VBucketAwareCouchbaseClient("http://{}:{}/pools/default".format(HOST, PORT), COUCHBASE_BUCKET)
document = client.get(COUCHBASE_DOCUMENT)
 
 
###############
# Version 0.7.1
###############
client = couchbase.Couchbase("{}:{}".format(HOST, PORT), USER, PASSWORD)
bucket = client.bucket(MEMCACHED_BUCKET) # Fails with BucketUnavailableException.
bucket = client.bucket(COUCHBASE_BUCKET) # Fails with BucketUnavailableException.
 
client = couchbase.couchbaseclient.CouchbaseClient("http://{}:{}/pools/default".format(HOST, PORT), MEMCACHED_BUCKET)
document = client.get(MEMCACHED_DOCUMENT)       # Fails with Exception: vbucket map does not have an entry for vb : -12782
document = client.memcached(MEMCACHED_DOCUMENT) # Fails with Exception: vbucket map does not have an entry for vb : -12782
 
# Fetching a document from a couchbase bucket succeeds.
client = couchbase.couchbaseclient.CouchbaseClient("http://{}:{}/pools/default".format(HOST, PORT), COUCHBASE_BUCKET)
document = client.get(COUCHBASE_DOCUMENT)

I'll see if the tests suggest any alternatives.

If you're getting BucketUnavailableExceptions there's likely something worse afoot.

I've recently found some bugs with the CouchbaseClient class that will be address sooner than later, but the BucketUnavailableExceptions shouldn't be happening if the server's running and the construction parameters (user name, password, etc) are setup properly.

The password in this case should be the cluster password, btw--not the bucket-level "sasl" password.

I'll keep digging...

I didn't realize that the Couchbase/Server class required the cluster password (but it makes sense now).

With the correct password, I get the same error I did with the CouchbaseClient class:

###############
# Version 0.7.1
###############
client = couchbase.Couchbase("{}:{}".format(HOST, PORT), USER, PASSWORD)
bucket = client.bucket(MEMCACHED_BUCKET)
document = bucket.get(MEMCACHED_DOCUMENT) # Fails with Exception: vbucket map does not have an entry for vb : -12782
 
# Fetching a document from a couchbase bucket succeeds.
bucket = client.bucket(COUCHBASE_BUCKET)
document = bucket.get(COUCHBASE_DOCUMENT)

Thanks for the update. I'll get some tests written here, and hopefully we can squash this one before the next release.

Benjamin,

Thanks for your help.

When I examine the JSON returned from our nodes, it appears all of our Memcached buckets are using the "ketama" node locator scheme, not "vbucket".

I've looked through most of the Python code, and it seems like everything assumes vbucket.

Is the Python client supposed support the ketama scheme? (If not, it would be helpful to throw a clear exception when "nodeLocator" isn't "vbucket".)

I've only just begun the process of making the Python SDK handle memcached buckets--hence the proliferation of vbucket stuff. Originally there was only one MemcachedClient and it only did vbucket based hashing. I'm splitting them in two (MemcachedClient & VBucketAwareClient), but 0.7 is the first very basic step in that direction.

Our primary focus is on the vbucket-based hashing support and membase/couchbase type buckets as those give the cluster the strongest elasticity.

What was the motivation to use a 'memcached' type bucket over a 'couchbase' one?

The only motivation for choosing a 'memcached' bucket type was because it contains transient data that doesn't need to be saved to disk.

Now that I've dug into this issue, I see the architecture differences between the 'memcached' and 'couchbase' types. Originally, I incorrectly thought they were identical except 'memcached' type skipped writing to disk.

-Jon-Eric

Makes sense. :) feel free to start a new thread here or ping me in #libcouchbase on irc.freenode.net if you have any other questions.

Any updates on this @BigBlueHat?

We are having the same problem and need memcached for caching. Thanks!
A

This is the giant hack I use to get around the issue:

import couchbase
 
def fix_client_memcached_buckets(client, memcachedclient):
    """
    Adapt client.Bucket so it can access memcached buckets.
    (By default, it assumes every bucket has vBuckets, which
    is only true for couchbase buckets, not memcached ones.)
    """
    class Bucket(client.Bucket):
        def __init__(self, name, server):
            super(Bucket, self).__init__(name, server)
 
            # Use the correct client for memcached buckets.
            if self.info.type == 'memcached':
                self.mc_client = memcachedclient.MemcachedClient(server.servers[0]['ip'])
                self.mc_client.sasl_auth_plain(self.info.name.encode('ascii'), self.info.saslPassword.encode('ascii'))
                self.mc_client.vbucket_count = 1
                self.mc_client.done = self.mc_client.close
    client.Bucket = Bucket
fix_client_memcached_buckets(couchbase.client, couchbase.memcachedclient)

Nice

Thanks for sharing.