{"id":1843,"date":"2014-12-16T17:36:08","date_gmt":"2014-12-16T17:36:08","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=1843"},"modified":"2014-12-16T17:36:08","modified_gmt":"2014-12-16T17:36:08","slug":"writing-your-own-storage-engine-memcached-part-2","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/","title":{"rendered":"Writing your own storage engine for Memcached, part 2"},"content":{"rendered":"

In the previous blog post I described the engine initialization and\u00a0destruction. This blog post will cover the memory allocation model\u00a0in the engine interface.<\/p>\n

The memcached core is responsible for allocating all of the\u00a0memory it needs for its connections (send \/ receive buffers\u00a0etc), and the engine is responsible for allocating (and\u00a0freeing) all of the memory it needs to keep track of the\u00a0items. The engines shouldn't have to care about the memory the\u00a0core allocates (and use), but the core will access the memory\u00a0managed by the engine.<\/p>\n

When the memcached core is about to store a new item it needs\u00a0to get a (as of today continous) buffer to store the data for the\u00a0item. The core will try to allocate this buffer by calling the\u00a0allocate<\/font><\/code> function in the API. So let's start\u00a0extending our example code by adding out own implementation\u00a0of the allocate<\/font><\/code> function. The first thing we need\u00a0to do is to add it to our engine descriptor we return from\u00a0create_instance<\/font><\/code>. We're going to add a number of\u00a0functions in todays entry, so let's just map all of them\u00a0while we're at it:<\/p>\n

\n
MEMCACHED_PUBLIC_API
ENGINE_ERROR_CODE create_instance(uint64_t interface,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 GET_SERVER_API get_server_api,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ENGINE_HANDLE **handle)
{
\u00a0 \u00a0[ … cut … ]
\u00a0 \/*
\u00a0 \u00a0* Map the API entry points to our functions that implement them.
\u00a0 \u00a0*\/
\u00a0 \u00a0h->engine.initialize = fs_initialize;
\u00a0 \u00a0h->engine.destroy = fs_destroy;
\u00a0 \u00a0h->engine.get_info = fs_get_info;
\u00a0 \u00a0h->engine.allocate = fs_allocate;
\u00a0 \u00a0h->engine.remove = fs_item_delete;
\u00a0 \u00a0h->engine.release = fs_item_release;
\u00a0 \u00a0h->engine.get = fs_get;
\u00a0 \u00a0h->engine.get_stats = fs_get_stats;
\u00a0 \u00a0h->engine.reset_stats = fs_reset_stats;
\u00a0 \u00a0h->engine.store = fs_store;
\u00a0 \u00a0h->engine.flush = fs_flush;
\u00a0 \u00a0h->engine.unknown_command = fs_unknown_command;
\u00a0 \u00a0h->engine.item_set_cas = fs_item_set_cas;
\u00a0 \u00a0h->engine.get_item_info = fs_get_item_info;<\/b>
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

The next thing we need to do is to create a data structure to\u00a0keep the information we need. The purpose of this tutorial\u00a0isn't to create a memory efficient implementation, but to\u00a0exercise\u00a0the API. So let's just create the following struct:<\/p>\n

\n
struct fs_item {
\u00a0 \u00a0void *key;
\u00a0 \u00a0size_t nkey;
\u00a0 \u00a0void *data;
\u00a0 \u00a0size_t ndata;
\u00a0 \u00a0int flags;
\u00a0 \u00a0rel_time_t exptime;
};
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

Our implementation of allocate<\/font><\/code> would then look\u00a0like:<\/p>\n

\u00a0<\/p>\n

\n
static ENGINE_ERROR_CODE fs_allocate(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const void* cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0item **item,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const void* key,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const size_t nkey,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const size_t nbytes,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const int flags,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const rel_time_t exptime)
{
\u00a0 \u00a0struct fs_item *it = malloc(sizeof(struct fs_item));
\u00a0 \u00a0if (it == NULL) {
\u00a0 \u00a0 \u00a0 return ENGINE_ENOMEM;
\u00a0 \u00a0}
\u00a0 \u00a0it->flags = flags;
\u00a0 \u00a0it->exptime = exptime;
\u00a0 \u00a0it->nkey = nkey;
\u00a0 \u00a0it->ndata = nbytes;
\u00a0 \u00a0it->key = malloc(nkey);
\u00a0 \u00a0it->data = malloc(nbytes);
\u00a0 \u00a0if (it->key == NULL || it->data == NULL) {
\u00a0 \u00a0 \u00a0 free(it->key);
\u00a0 \u00a0 \u00a0 free(it->data);
\u00a0 \u00a0 \u00a0 free(it);
\u00a0 \u00a0 \u00a0 return ENGINE_ENOMEM;
\u00a0 \u00a0}
\u00a0 \u00a0memcpy(it->key, key, nkey);
\u00a0 \u00a0*item = it;
\u00a0 \u00a0return ENGINE_SUCCESS;
}
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

If you look in the implementation above you'll see that we didn't\u00a0return the pointer to the actual memory for the data storage to\u00a0the memcached core. To get that address memcached will call\u00a0get_item_info<\/font><\/code> in the API. So let's implememnt that:<\/p>\n

\n
static bool fs_get_item_info(ENGINE_HANDLE *handle, const void *cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0const item* item, item_info *item_info)
{
\u00a0 \u00a0struct fs_item* it = (struct fs_item*)item;
\u00a0 \u00a0if (item_info->nvalue < 1) {
\u00a0 \u00a0 \u00a0 return false;
\u00a0 \u00a0}<\/p>\n

\u00a0 \u00a0item_info->cas = 0; \/* Not supported *\/
\u00a0 \u00a0item_info->clsid = 0; \/* Not supported *\/
\u00a0 \u00a0item_info->exptime = it->exptime;
\u00a0 \u00a0item_info->flags = it->flags;
\u00a0 \u00a0item_info->key = it->key;
\u00a0 \u00a0item_info->nkey = it->nkey;
\u00a0 \u00a0item_info->nbytes = it->ndata; \/* Total length of the items data *\/
\u00a0 \u00a0item_info->nvalue = 1; \/* Number of fragments used *\/
\u00a0 \u00a0item_info->value[0].iov_base = it->data; \/* pointer to fragment 1 *\/
\u00a0 \u00a0item_info->value[0].iov_len = it->ndata; \/* Length of fragment 1 *\/<\/p>\n

\u00a0 \u00a0return true;
}
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

The get_item_info<\/font><\/code> function is important and\u00a0deserve more information. If you look in the engine API the\u00a0“item” is defined as a void pointer, and we defined our own\u00a0item-structure to keep track of the information we need on a\u00a0per item basis. The memcached core will however need to know
where to read \/ write the memory for the key and the data\u00a0going to \/ coming from a clinet. To do so in will invoke\u00a0get_item_info<\/font><\/code>. If you look closely at our\u00a0implementation of fs_get_item_info<\/font><\/code> you will see\u00a0that the first thing I'm doing is to check that\u00a0item_info->nvalue<\/font><\/code> contains at least 1
element. Right now it will always<\/b> be one, but the\u00a0intention is that we're going to support scattered IO.<\/p>\n

When the core is done moving the data it received over the wire\u00a0into the item, it will try to store the item in our engine by\u00a0calling store<\/font><\/code>. So let's go ahead and create a simple\u00a0implementation (we'll extend it later on in the tutorial):<\/p>\n

\n
static ENGINE_ERROR_CODE fs_store(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void *cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 item* item,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 uint64_t *cas,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ENGINE_STORE_OPERATION operation,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 uint16_t vbucket)
{
\u00a0 \u00a0struct fs_item* it = item;
\u00a0 \u00a0char fname[it->nkey + 1];
\u00a0 \u00a0memcpy(fname, it->key, it->nkey);
\u00a0 \u00a0fname[it->nkey] = ‘\u0000’;
\u00a0 \u00a0FILE *fp = fopen(fname, “w”);
\u00a0 \u00a0if (fp == NULL) {
\u00a0 \u00a0 \u00a0 return ENGINE_NOT_STORED;
\u00a0 \u00a0}
\u00a0 \u00a0size_t nw = fwrite(it->data, 1, it->ndata, fp);
\u00a0 \u00a0fclose(fp);
\u00a0 \u00a0if (nw != it->ndata) {
\u00a0 \u00a0 \u00a0 remove(fname);
\u00a0 \u00a0 \u00a0 return ENGINE_NOT_STORED;
\u00a0 \u00a0}<\/p>\n

\u00a0 \u00a0*cas = 0;
\u00a0 \u00a0return ENGINE_SUCCESS;
}
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

If you look at the implementation above you will see that it\u00a0doesn't implement the correct semantics\u00a0for add<\/em>\/replace<\/em>\/set<\/em> etc, and it will\u00a0block memcached while we're doing file IO. Don't care about that\u00a0right now, because we'll get back to that.<\/p>\n

When the core is done using the item it allocated, it will release\u00a0the item by calling the release<\/font><\/code> function in the API.\u00a0The engine may reuse the items storage for something else at this\u00a0time. So let's hook up our release<\/font><\/code> implementation:<\/p>\n

\n
static void fs_item_release(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void *cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 item* item)
{
\u00a0 \u00a0struct fs_item *it = item;
\u00a0 \u00a0free(it->key);
\u00a0 \u00a0free(it->data);
\u00a0 \u00a0free(it);
}
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

Now we've created all of the code to successfully store items\u00a0in our engine, but we can't read any of them back. So let's\u00a0implement get<\/code><\/font><\/p>\n

\n
static ENGINE_ERROR_CODE fs_get(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 item** item,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* key,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const int nkey,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 uint16_t vbucket)
{<\/p>\n

\u00a0 \u00a0char fname[nkey + 1];
\u00a0 \u00a0memcpy(fname, key, nkey);
\u00a0 \u00a0fname[nkey] = ‘\u0000’;<\/p>\n

\u00a0 \u00a0struct stat st;
\u00a0 \u00a0if (stat(fname, &st) == -1) {
\u00a0 \u00a0 \u00a0 return ENGINE_NOT_STORED;
\u00a0 \u00a0}<\/p>\n

\u00a0 \u00a0struct fs_item* it = NULL;
\u00a0 \u00a0ENGINE_ERROR_CODE ret = fs_allocate(handle, cookie, (void**)&it, key, nkey,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0st.st_size, 0, 0);
\u00a0 \u00a0if (ret != ENGINE_SUCCESS) {
\u00a0 \u00a0 \u00a0 return ENGINE_ENOMEM;
\u00a0 \u00a0}<\/p>\n

\u00a0 \u00a0FILE *fp = fopen(fname, “r”);
\u00a0 \u00a0if (fp == NULL) {
\u00a0 \u00a0 \u00a0 fs_release(handle, cookie, it);
\u00a0 \u00a0 \u00a0 return ENGINE_FAILED;
\u00a0 \u00a0}<\/p>\n

\u00a0 \u00a0size_t nr = fread(it->data, 1, it->ndata, fp);
\u00a0 \u00a0fclose(fp);
\u00a0 \u00a0if (nr != it->ndata) {
\u00a0 \u00a0 \u00a0 fs_release(handle, cookie, it);
\u00a0 \u00a0 \u00a0 return ENGINE_FAILED;
\u00a0 \u00a0}<\/p>\n

\u00a0 \u00a0*item = it;
\u00a0 \u00a0return ENGINE_SUCCESS;
}
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

Let's add a dummy implementation for the rest of the API and\u00a0try to load and test the engine:<\/p>\n

\n
static const engine_info* fs_get_info(ENGINE_HANDLE* handle)
{
\u00a0 \u00a0static engine_info info = {
\u00a0 \u00a0 \u00a0 .description = “Filesystem engine v0.1”,
\u00a0 \u00a0 \u00a0 .num_features = 0
\u00a0 \u00a0};<\/p>\n

\u00a0 \u00a0return &info;
}<\/p>\n

static ENGINE_ERROR_CODE fs_item_delete(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* key,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const size_t nkey,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 uint64_t cas,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 uint16_t vbucket)
{
\u00a0 \u00a0return ENGINE_KEY_ENOENT;
}<\/p>\n

static ENGINE_ERROR_CODE fs_get_stats(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const char* stat_key,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 int nkey,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ADD_STAT add_stat)
{
\u00a0 \u00a0return ENGINE_SUCCESS;
}<\/p>\n

static ENGINE_ERROR_CODE fs_flush(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* cookie, time_t when)
{<\/p>\n

\u00a0 \u00a0return ENGINE_SUCCESS;
}<\/p>\n

static void fs_reset_stats(ENGINE_HANDLE* handle, const void *cookie)
{<\/p>\n

}<\/p>\n

static ENGINE_ERROR_CODE fs_unknown_command(ENGINE_HANDLE* handle,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 const void* cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 protocol_binary_request_header *request,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ADD_RESPONSE response)
{
\u00a0 \u00a0return ENGINE_ENOTSUP;
}<\/p>\n

static void fs_item_set_cas(ENGINE_HANDLE *handle, const void *cookie,
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 item* item, uint64_t val)
{
}<\/p>\n

\u00a0 \u00a0 \u00a0 <\/p><\/div>\n<\/div>\n

So let's go ahead and try our engine:<\/p>\n

\n
trond@opensolaris> \/opt\/memcached\/bin\/memcached -E .libs\/fs_engine.so
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

From another terminal I'm typing in:<\/p>\n

\n
trond@opensolaris> telnet localhost 11211
Trying ::1…
Connected to opensolaris.
Escape character is ‘^]’.
add test 0 0 4
test
STORED
get test
VALUE test 0 4
test
END
quit
Connection to storm closed by foreign host.
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

Terminate memcached by pressing ctrl-c<\/font><\/code>, and look in the current directory:<\/p>\n

\n
trond@opensolaris> ls -l test
-rw-r–r– \u00a0 1 trond \u00a0 \u00a0users \u00a0 \u00a0 \u00a0 \u00a0 \u00a06 Oct \u00a08 12:56 test
trond@opensolaris> cat test
test
\u00a0 \u00a0 \u00a0 <\/div>\n<\/div>\n

That's all for this time.<\/p>\n","protected":false},"excerpt":{"rendered":"

In the previous blog post I described the engine initialization and\u00a0destruction. This blog post will cover the memory allocation model\u00a0in the engine interface. The memcached core is responsible for allocating all of the\u00a0memory it needs for its connections (send \/ […]<\/p>\n","protected":false},"author":14,"featured_media":13873,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"ppma_author":[8981],"class_list":["post-1843","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"acf":[],"yoast_head":"\nWriting your own storage engine for Memcached, part 2 - The Couchbase Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Writing your own storage engine for Memcached, part 2\" \/>\n<meta property=\"og:description\" content=\"In the previous blog post I described the engine initialization and\u00a0destruction. This blog post will cover the memory allocation model\u00a0in the engine interface. The memcached core is responsible for allocating all of the\u00a0memory it needs for its connections (send \/ […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-12-16T17:36:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2022\/11\/couchbase-nosql-dbaas.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Trond Norbye, Senior Developer, Couchbase\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Trond Norbye, Senior Developer, Couchbase\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\"},\"author\":{\"name\":\"Trond Norbye, Senior Developer, Couchbase\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/ef0f9ff42d878c2fc0ab3a685e96f36d\"},\"headline\":\"Writing your own storage engine for Memcached, part 2\",\"datePublished\":\"2014-12-16T17:36:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\"},\"wordCount\":1384,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\",\"name\":\"Writing your own storage engine for Memcached, part 2 - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"datePublished\":\"2014-12-16T17:36:08+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"width\":1800,\"height\":630},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Writing your own storage engine for Memcached, part 2\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/ef0f9ff42d878c2fc0ab3a685e96f36d\",\"name\":\"Trond Norbye, Senior Developer, Couchbase\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/03d13b4ab5eaa14c91cab7658f04df07\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/bf3c00ca228efa25f7bfc168f566d6389279b44d4bbba4683c260a8bf33da03d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/bf3c00ca228efa25f7bfc168f566d6389279b44d4bbba4683c260a8bf33da03d?s=96&d=mm&r=g\",\"caption\":\"Trond Norbye, Senior Developer, Couchbase\"},\"description\":\"Trond Norbye is a Software Architect at Couchbase. Core contributor to Couchbase & Memcached projects. Created the C\/C++ & node.js Couchbase client libraries.\",\"url\":\"https:\/\/www.couchbase.com\/blog\/author\/trond-norbye\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Writing your own storage engine for Memcached, part 2 - The Couchbase Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/","og_locale":"en_US","og_type":"article","og_title":"Writing your own storage engine for Memcached, part 2","og_description":"In the previous blog post I described the engine initialization and\u00a0destruction. This blog post will cover the memory allocation model\u00a0in the engine interface. The memcached core is responsible for allocating all of the\u00a0memory it needs for its connections (send \/ […]","og_url":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/","og_site_name":"The Couchbase Blog","article_published_time":"2014-12-16T17:36:08+00:00","og_image":[{"width":1800,"height":630,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2022\/11\/couchbase-nosql-dbaas.png","type":"image\/png"}],"author":"Trond Norbye, Senior Developer, Couchbase","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Trond Norbye, Senior Developer, Couchbase","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/"},"author":{"name":"Trond Norbye, Senior Developer, Couchbase","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/ef0f9ff42d878c2fc0ab3a685e96f36d"},"headline":"Writing your own storage engine for Memcached, part 2","datePublished":"2014-12-16T17:36:08+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/"},"wordCount":1384,"commentCount":0,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/","url":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/","name":"Writing your own storage engine for Memcached, part 2 - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","datePublished":"2014-12-16T17:36:08+00:00","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","width":1800,"height":630},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/writing-your-own-storage-engine-memcached-part-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Writing your own storage engine for Memcached, part 2"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/ef0f9ff42d878c2fc0ab3a685e96f36d","name":"Trond Norbye, Senior Developer, Couchbase","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/03d13b4ab5eaa14c91cab7658f04df07","url":"https:\/\/secure.gravatar.com\/avatar\/bf3c00ca228efa25f7bfc168f566d6389279b44d4bbba4683c260a8bf33da03d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/bf3c00ca228efa25f7bfc168f566d6389279b44d4bbba4683c260a8bf33da03d?s=96&d=mm&r=g","caption":"Trond Norbye, Senior Developer, Couchbase"},"description":"Trond Norbye is a Software Architect at Couchbase. Core contributor to Couchbase & Memcached projects. Created the C\/C++ & node.js Couchbase client libraries.","url":"https:\/\/www.couchbase.com\/blog\/author\/trond-norbye\/"}]}},"authors":[{"term_id":8981,"user_id":14,"is_guest":0,"slug":"trond-norbye","display_name":"Trond Norbye, Senior Developer, Couchbase","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/bf3c00ca228efa25f7bfc168f566d6389279b44d4bbba4683c260a8bf33da03d?s=96&d=mm&r=g","author_category":"","last_name":"Norbye","first_name":"Trond","job_title":"","user_url":"","description":"Trond Norbye is a Software Architect at Couchbase. Core contributor to Couchbase & Memcached projects. Created the C\/C++ & node.js Couchbase client libraries."}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/1843","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=1843"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/1843\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/13873"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=1843"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=1843"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=1843"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}