problems with libcouchbase lcb_store
I am trying to wrap libcouchbase for use from python and have run into a snag with lcb_store. I am able to store a document to couchbase, but when viewed through the couchbase server GUI, the document is not ascii text. In addition, I have not been able to get an LCB_SUCCESS error value passed to my store_callback.
I am using libcouchbase 2.0.4
I have taken sample code directly from Trond's blog but am having the same problems. My most recent experiments were run with the following function and callback:
static const char const *key = "mykey";
static const char const *value = "myvalue";
static void store_key(lcb_t instance)
{
lcb_store_cmd_t cmd;
lcb_error_t error;
const lcb_store_cmd_t * const commands[] = { &cmd };
memset(&cmd, 0, sizeof(cmd));
cmd.v.v0.key = key;
cmd.v.v0.nkey = strlen(key);
cmd.v.v0.bytes = value;
cmd.v.v0.nbytes = strlen(value);
cmd.v.v0.operation = LCB_SET;
if ((error = lcb_store(instance, NULL, 1, commands)) != LCB_SUCCESS) {
fprintf(stderr, "Failed to store key: %s\n",
lcb_strerror(instance, error));
exit(EXIT_FAILURE);
}
}
static void
store_callback(lcb_t instance, const void *cookie,
lcb_error_t error, lcb_store_resp_t *resp)
{
if (error )
fprintf(stdout, "ERROR: %s (0x%x)\n",
lcb_strerror(instance, error), error);
}I varied "cmd.v.v0.operation" with the following results:
for cmd.v.v0.operation = LCB_SET (0x03)"
- store_callback prints: "ERROR: Not a number (0x3)"
- couchbase GUI shows: "bXl2YWx1ZQ=="
- lcb_get returns: " key:mykey, document:myvalue"
for cmd.v.v0.operation = LCB_APPEND (0x04)
- store_callback prints: "ERROR: Object too big (0x4)"
- couchbase GUI shows: "bXl2YWx1ZW15dmFsdWU="
- lcb_get returns: key:mykey, document:myvaluemyvalue
for cmd.v.v0.operation = LCB_PREPEND (0x05)
- store_callback prints: "ERROR: Too busy. Try again later (0x5)"
- couchbase GUI shows: "bXl2YWx1ZW15dmFsdWVteXZhbHVl"
- lcb_get returns: key:mykey, document:myvaluemyvaluemyvalue
for cmd.v.v0.operation = LCB_APPEND (0x01)
- store_callback prints: "ERROR: Continue authentication (0x1)"
- couchbase GUI shows: "bXl2YWx1ZW15dmFsdWVteXZhbHVl"
- lcb_get returns: key:mykey, document:myvaluemyvaluemyvalue
for cmd.v.v0.operation = LCB_REPLACE (0x02)
- store_callback prints: "ERROR: Authentication error (0x2)"
- couchbase GUI shows: "bXl2YWx1ZQ=="
- lcb_get returns: key:mykey, document:myvalue
As you can see, it kind of does the right thing, but returns an error code equal to the operation field, and appears to be saving the document with a non-ascii encoding. Clear as mud?
Thanks for the JSON tip. I think I knew that once upon a time, but forgot. I jsonned up my document a bit better and now it looks good in the web ui.
As far as the full source & setup goes, I was hoping to spare you. I'm in the early stages of building a python wrapper for libcouchbase, so the code is full of PyObjects and PyArg_ParseTuples and the like. Hopefully you can see past all that to the underlying lcb code. I took out the sample and reverted to the python extension in all its glory. I'm having the same problems either way.
First the python extension library:
#include <Python.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <libcouchbase/couchbase.h>
void
lcb_instance_destructor(PyObject *capsule) {
lcb_t *instance;
instance = PyCapsule_GetPointer(capsule, "lcb_instance");
free(instance);
}
/* ----------------------------------------
"error" callback handlers
---------------------------------------- */
static PyObject *py_error_callback = NULL;
static PyObject *
_set_error_callback(PyObject *self, PyObject *args)
{
PyObject *temp;
if (!PyArg_ParseTuple(args, "O:set_error_callback", &temp))
return NULL;
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(py_error_callback); /* Dispose of previous callback */
py_error_callback = temp; /* Remember new callback */
Py_INCREF(Py_None);
return Py_None;
}
static void
error_callback(lcb_t instance, lcb_error_t error, const char *errinfo)
{
PyObject *arglist;
PyObject *result;
if (py_error_callback) {
arglist = Py_BuildValue("is", error, errinfo);
if (!arglist)
return;
result = PyObject_CallObject(py_error_callback, arglist);
Py_DECREF(arglist);
if (result)
Py_DECREF(result);
}
}
/* ----------------------------------------
"get" callback handlers
---------------------------------------- */
static PyObject *py_get_callback = NULL;
static PyObject *
_set_get_callback(PyObject *self, PyObject *args)
{
PyObject *temp;
if (!PyArg_ParseTuple(args, "O:set_get_callback", &temp))
return NULL;
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(py_get_callback); /* Dispose of previous callback */
py_get_callback = temp; /* Remember new callback */
Py_INCREF(Py_None);
return Py_None;
}
static void
get_callback(lcb_t instance, const void *cookie,
lcb_error_t error, lcb_get_resp_t *resp)
{
PyObject *arglist;
PyObject *result;
if (py_get_callback) {
arglist = Py_BuildValue("is#s#", error, resp->v.v0.key, resp->v.v0.nkey, resp->v.v0.bytes, resp->v.v0.nbytes);
if (!arglist)
return;
result = PyObject_CallObject(py_get_callback, arglist);
Py_DECREF(arglist);
if (result)
Py_DECREF(result);
}
}
/* ----------------------------------------
"store" callback handlers
---------------------------------------- */
static PyObject *py_store_callback = NULL;
static PyObject *
_set_store_callback(PyObject *self, PyObject *args)
{
PyObject *temp;
if (!PyArg_ParseTuple(args, "O:set_store_callback", &temp))
return NULL;
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(py_store_callback); /* Dispose of previous callback */
py_store_callback = temp; /* Remember new callback */
Py_INCREF(Py_None);
return Py_None;
}
static void
store_callback(lcb_t instance, const void *cookie,
lcb_error_t error, lcb_store_resp_t *resp)
{
PyObject *arglist;
PyObject *result;
fprintf(stdout, "store_callback, arguments: instance:%p, cookie:%p "
"error:%d, resp:%p\n", instance, cookie, error, resp);
if (py_store_callback) {
if (error == LCB_SUCCESS) {
arglist = Py_BuildValue("is#", error, resp->v.v0.key, resp->v.v0.nkey);
} else {
arglist = Py_BuildValue("is", error, NULL);
}
if (!arglist)
return;
result = PyObject_CallObject(py_store_callback, arglist);
Py_DECREF(arglist);
if (result)
Py_DECREF(result);
}
}
static PyObject *
_connect(PyObject *self, PyObject *args) {
lcb_error_t err;
struct lcb_create_st create_options;
struct lcb_create_io_ops_st io_opts;
char errMsg[256];
char *host = NULL;
char *user = NULL;
char *passwd = NULL;
char *bucket = NULL;
if (!PyArg_ParseTuple(args, "|ssss", &host, &user, &passwd, &bucket))
return NULL;
io_opts.version = 0;
io_opts.v.v0.type = LCB_IO_OPS_DEFAULT;
io_opts.v.v0.cookie = NULL;
err = lcb_create_io_ops(&create_options.v.v0.io, &io_opts);
if (err != LCB_SUCCESS) {
snprintf(errMsg, 256, "Failed to create IO instance: %s\n",
lcb_strerror(NULL, err));
PyErr_SetString(PyExc_IOError, errMsg);
return NULL;
}
memset(&create_options, 0, sizeof(create_options));
if (host) {
create_options.v.v0.host = host;
}
if (user) {
create_options.v.v0.user = user;
}
if (passwd) {
create_options.v.v0.passwd = passwd;
}
if (bucket) {
create_options.v.v0.bucket = bucket;
}
lcb_t *instance = calloc(1, sizeof(lcb_t));
if (!instance) {
PyErr_SetString(PyExc_MemoryError, "ran out of memory while allocating lcb_t instance");
return NULL;
}
fprintf (stdout, "connecting, host:%s, user:%s, passwd:%s, bucket:%s\n",
host, user, passwd, bucket);
err = lcb_create(instance, &create_options);
if (err != LCB_SUCCESS) {
free(instance);
snprintf(errMsg, 256, "Failed to create libcouchbase instance: %s\n",
lcb_strerror(NULL, err));
PyErr_SetString(PyExc_IOError, errMsg);
}
fprintf(stdout, "created lcb_instance @%p\n", (void *) *instance);
(void)lcb_set_error_callback(*instance, (lcb_error_callback) error_callback);
/* Initiate the connect sequence in libcouchbase */
fprintf(stdout, "connecting with lcb_instance @%p\n", (void *) *instance);
/* Initiate the connect sequence in libcouchbase */
if ((err = lcb_connect(*instance)) != LCB_SUCCESS) {
snprintf(errMsg, 256, "Failed to initiate connect: %s\n",
lcb_strerror(NULL, err));
PyErr_SetString(PyExc_IOError, errMsg);
lcb_destroy(*instance);
return NULL;
}
(void)lcb_set_get_callback(*instance, (lcb_get_callback) get_callback);
(void)lcb_set_store_callback(*instance, (lcb_store_callback) store_callback);
return PyCapsule_New(instance, "lcb_instance", lcb_instance_destructor);
}
static PyObject *
_get(PyObject *self, PyObject *args) {
PyObject *capsule;
char *key = NULL;
lcb_t *instance;
lcb_get_cmd_t cmd;
const lcb_get_cmd_t *commands[1];
lcb_error_t err;
char errMsg[256];
if (!PyArg_ParseTuple(args, "Os", &capsule, &key)) {
return NULL;
}
instance = PyCapsule_GetPointer(capsule, "lcb_instance");
commands[0] = &cmd;
memset(&cmd, 0, sizeof(cmd));
cmd.v.v0.key = key;
cmd.v.v0.nkey = strlen(key);
err = lcb_get(*instance, NULL, 1, commands);
if (err != LCB_SUCCESS) {
snprintf(errMsg, 256, "Failed to get: %s\n",
lcb_strerror(NULL, err));
PyErr_SetString(PyExc_IOError, errMsg);
return NULL;
}
Py_INCREF(Py_None);
return Py_None;
}
static PyObject *
_store(PyObject *self, PyObject *args) {
PyObject *capsule;
char *key;
char *value;
int operation;
lcb_t *instance;
lcb_store_cmd_t cmd;
const lcb_store_cmd_t *commands[1];
lcb_error_t err;
char errMsg[256];
if (!PyArg_ParseTuple(args, "Ossi", &capsule, &key, &value, &operation)) {
return NULL;
}
instance = PyCapsule_GetPointer(capsule, "lcb_instance");
memset(&cmd, 0, sizeof(cmd));
cmd.v.v0.key = key;
cmd.v.v0.nkey = strlen(key);
cmd.v.v0.bytes = value;
cmd.v.v0.nbytes = strlen(value);
cmd.v.v0.operation = operation;
commands[0] = &cmd;
fprintf(stdout, "key:%s\nnkey:%ld\nbytes:%s\nnbytes:%ld\noperation:%d\n"
"flags:%x\ncas:%lld\nexptime:%ld\nversion:%d\n",
cmd.v.v0.key, cmd.v.v0.nkey, cmd.v.v0.bytes, cmd.v.v0.nbytes,
cmd.v.v0.operation, cmd.v.v0.flags, cmd.v.v0.cas,
cmd.v.v0.exptime, cmd.version);
err = lcb_store(*instance, NULL, 1, commands);
if (err != LCB_SUCCESS) {
snprintf(errMsg, 256, "Failed to store: %s\n",
lcb_strerror(NULL, err));
PyErr_SetString(PyExc_IOError, errMsg);
return NULL;
}
Py_INCREF(Py_None);
return Py_None;
}
static PyObject *
_wait(PyObject *self, PyObject *args) {
PyObject *capsule;
lcb_t *instance;
if (!PyArg_ParseTuple(args, "O", &capsule))
return NULL;
instance = PyCapsule_GetPointer(capsule, "lcb_instance");
lcb_wait(*instance);
Py_INCREF(Py_None);
return Py_None;
}
static PyMethodDef
LcbMethods[] = {
{ "set_error_callback", _set_error_callback, METH_VARARGS,
"Set callback for error"},
{ "set_get_callback", _set_get_callback, METH_VARARGS,
"Set callback for get"},
{ "set_store_callback", _set_store_callback, METH_VARARGS,
"Set callback for store"},
{ "connect", _connect, METH_VARARGS,
"Connect to a Couchbase cluster" },
{ "get", _get, METH_VARARGS,
"Get a key from Couchbase cluster" },
{ "store", _store, METH_VARARGS,
"Store a key in Couchbase cluster" },
{ "wait", _wait, METH_VARARGS,
"wait for couchbase call to complete" },
{ NULL, NULL, 0, NULL }
};
PyMODINIT_FUNC
initpylcb(void)
{
(void) Py_InitModule("pylcb", LcbMethods);
}Now the python test code:
import pylcb
import json
def error_callback(error, errinfo):
print "error in pylcb: %x, %s" % (error, errinfo)
def get_callback(error, key, document):
print "callback for pylcb.get, error:%x, key:%s, document:%s" % \
(error, key, document)
def store_callback(error, key):
print "callback for pylcb.store, error:%x, key:%s" % (error, key)
pylcb.set_error_callback(error_callback)
pylcb.set_get_callback(get_callback)
pylcb.set_store_callback(store_callback)
print "----- connect -----"
instance = pylcb.connect("localhost", "Administrator", "password", "jobs")
pylcb.wait(instance)
print "\n----- store -----"
test = dict(frank="is awesome", computers="are not")
testJson = json.dumps(test, indent=4)
pylcb.store(instance, "test", testJson, 3)
pylcb.wait(instance)
print "\n----- get -----"
pylcb.get(instance, "test")
pylcb.wait(instance)and lastly the results of running the python test:
----- connect -----
connecting, host:localhost, user:Administrator, passwd:password, bucket:jobs
created lcb_instance @0x7ff5db877a00
connecting with lcb_instance @0x7ff5db877a00
----- store -----
key:test
nkey:4
bytes:{
"frank": "is awesome",
"computers": "are not"
}
nbytes:58
operation:3
flags:0
cas:0
exptime:0
version:0
store_callback, arguments: instance:0x7ff5db877a00, cookie:0x0 error:3, resp:0x0
callback for pylcb.store, error:3, key:None
----- get -----
callback for pylcb.get, error:0, key:test, document:{
"frank": "is awesome",
"computers": "are not"
}The current plan for the Python SDK [1] is to use libcouchbase. For the wrapping the C API Cython [2] will be used. I've already patched a tool to auto-generate the needed files for Cython, so you can basically use libcouchbase straight away.
If you like to help out, join the couchbase mailing list/Google Group so we can stay in touch.
[1] https://groups.google.com/group/couchbase/browse_thread/thread/1ab853690...
[2] http://cython.org/
Cheers,
Volker
If you like to help out, join the couchbase mailing list/Google Group so we can stay in touch.
[1] https://groups.google.com/group/couchbase/browse_thread/thread/1ab853690...
[2] http://cython.org/
Cheers,
Volker
Thanks, Volker. I typed a short post in the Google Group about what I'm up to. For the moment I'd like to continue on my current path, but I will also keep and eye on and try to contribute to your new effort. Couchbase badly needs a new & improved python client.
I am baffled by my problem as posted above. With my JSON-stupidity cleared up, it's obvious that libcouchbase is performing the store as instructed, but somehow I'm getting the operation field back as an error in the callback. I've double/triple/quadruple-checked my callback syntax, but it sure looks right to me.
I forgot to mention - this is all running on Mac OS X (currently version 10.8.3). Once it's working it will be deployed on ubuntu linux. I also tried with libcouchbase 2.0.1 with the same results.
One last note - I used tcpdump to take a look at the traffic on port 11210. If I'm not mistaken, these two packets are the set request & response. I know next to nothing of the memcached binary protocol, but I think they look OK.
Request:
--------
0x0000: 4500 0092 db13 4000 4006 0000 7f00 0001 E.....@.@.......
0x0010: 7f00 0001 c29e 2bca ad31 2ed1 2533 1008 ......+..1..%3..
0x0020: 8018 23e0 fe86 0000 0101 080a 389d 01b9 ..#.........8...
0x0030: 389d 01b9 8001 0004 0800 003f 0000 0046 8..........?...F
0x0040: 0100 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 0000 7465 7374 7b0a 2020 2020 2266 ....test{....."f
0x0060: 7261 6e6b 223a 2022 6973 2061 7765 736f rank":."is.aweso
0x0070: 6d65 222c 200a 2020 2020 2263 6f6d 7075 me",......"compu
0x0080: 7465 7273 223a 2022 6172 6520 6e6f 7422 ters":."are.not"
0x0090: 0a7d .}
Response:
--------
0x0000: 4500 004c 4d7b 4000 4006 0000 7f00 0001 E..LM{@.@.......
0x0010: 7f00 0001 2bca c29e 2533 1008 ad31 2f2f ....+...%3...1//
0x0020: 8018 23da fe40 0000 0101 080a 389d 01b9 ..#..@......8...
0x0030: 389d 01b9 8101 0000 0000 0000 0000 0000 8...............
0x0040: 0100 0000 0000 8d7c 772f a7a2 .......|w/..I figured it out.
the manpage documentation for lcb_set_store_callback is wrong. On GitHub at: https://github.com/couchbase/libcouchbase/blob/master/man/man3couchbase/...
in particular, it shows the arguments for the lcb_store_callback function as follows:
void function(lcb_t instance,
const void* cookie,
lcb_error_t error,
lcb_store_resp_t * resp);This is incorrect. It is properly prototyped in "callbacks.h"
typedef void (*lcb_store_callback)(lcb_t instance,
const void *cookie,
lcb_storage_t operation,
lcb_error_t error,
const lcb_store_resp_t *resp);I think that all of the examples that I looked at contain the same error.
the web ui is showing you plain text only if you are storing JSON, otherwise it thinks that you are storing blob and uses base64 to display it. your other errors look strange. could you attach full source and describe your setup and the way how are you executing the sample
Find me on FreeNode IRC in #libcouchbase channel