Common Utilities for C C++ SDK functionality

Skip to end of metadata
Go to start of metadata

This page will discuss some of the work required to unify and consolidate common tasks which are required in order for libcouchbase to function effectively in user environments.

As it stands, libcouchbase is primarily the protocol layer (both memcached and HTTP) for talking to the Couchbase server; however there are some higher level operations which may need further common handling:

Functional Requirements/Specifications

Durability API

This section will discuss the durability API which may be provided in a future utility library. The API should be neutral enough to be integrated into all libcouchbase-based SDKs.

It should provide an API to allow for the verification of a specific durability criteria across one or many nodes, and should be event-based:

API Inputs

Should follow the general format of an lcb_* command structure.

  • Key
  • Expected CAS
  • Number of servers to which the key should be persisted (or deleted) to
  • Number of servers to which the key should be replicated (or deleted) to
  • Poll interval (for retries between cycles)
  • Timeout interval (after which a response will be received indicating the durability requirements have not been satisfied)
  • Sense. This means whether the request is for a negative verification (as in the case of deletes), or positive verification (as is the case of mutations)

API Outputs

API Should follow the form of a lcb_*_resp_t. The callback should be invoked for the requested key (in the input) after either the durability requirement has been satisfied, or when it has been determined that the requirements may never be satisfied (e.g. different CAS on master), or when the timeout interval has expired.

  • Key
  • CAS (as reported by server)
  • Number of servers key was persisted to
  • Number of servers key was replicated to
  • Error code (this will be an lcb_error_t, either indicating success, or a reason for which the durability could not be satisfied)

Usage Model

This operation will present itself as a normal libcouchbase operation and will occupy a refcount in the event loop (that is, lcb_wait should not return until all pending "durability" operations have been completed.

The user should be able to schedule this operation either from a successful set/delete callback, OR as an individual wait sequence after lcb_wait. In either event the handling should be fairly opaque to the user.

Example Code

This code is from a WIP changeset in gerrit, but contains an outline and example of how it may be used:

View Parameter API

This API is predominantly a string library. It allows a consistent and symbolic way of handling and creating a Couchbase view query string. Its primary purpose is to provide a common interface and behavior for converting between "classical" and equivalent semantics to strict parameters as expected by the View REST API.

API Inputs

  • View Name
  • Design Document Name
  • List of query parameters. Each query parameter may be created using the following inputs:
    • String parameter (i.e. "group_level") OR a symbolic constant, e.g. LCB_VOPT_GROUPLEVEL
    • Value Type. This may be either an int, string, boolean, etc. - dependent on the parameter name. Each parameter may have a callback to verify and convert to the appropriate value
    • Value Conversion Options; these include:
      • Whether to validate the value
      • Whether to URI-escape the value
      • Whether the value pointer should be copied as an internal string
      • Whether to use "ad-hoc" mode (i.e. both parameter and value are user-supplied. Useful for supporting new options in the server without requiring an update to the SDK code)

API Outputs

As this is a string library, the output will always be a simple string. This string may be optionally URI-encoded but should be configurable (as it may be the case that the user is already passing a URI-encoded string). The string shall be allocated and free-able by using free(3)

Usage Model

This library feature is intended to be used in conjunction with other SDK-native features and functions. Specifically it intends to eliminate the option processing code which is fairly duplicate across all SDKs.

There may still be need to actually use a native JSON library to encode JSON parameters. How this is handled is SDK specific.

Example Code

As it exists in php-ext-couchbase

And a slightly-modified version, with some fixes:

View Row API

This API is intended to help with common pre-processing and buffering of view responses. This is particularly useful for large payloads. It also provides an option to incrementally fetch the views.

API Inputs

The view API inputs consist of feeding string data into an opaque parser context. The parser context will also be set up with callbacks which will be invoked upon receipt of each parsed view row, or on a parse error.

  • Opaque parse context (API function to initialize/create this)
  • Error callback
  • Row callback

API Output

  • Row Callback
    • Will contain a single row only
  • Error Callback
    • Will contain an error code, as well as the raw text for the response
  • Parse context
    • Will contain non-row meta-data returned in the view result, as well as any HTTP metadata (such as the return code)

Usage Model

The usage model for the incremental row parsing is slightly more involved as it requires setting up two different callbacks. Generally speaking however, the receipt of each view row is to follow the semantics of responses for something like a multi-get, in that a new callback is invoked for each row.

One is required to create a "view row parser context" (which may be re-used) which allocates the necessary handles for the parser object. Then incoming data is fed into the buffer (a convenience HTTP callback may be provided to allow for this boilerplate to be performed transparently). When sufficient data has been fed and at least one row has been internally delimited by the parser, the row callback is invoked. If an error occurs during parsing, the error callback is invoked.

Example Code

Common Flag/Datatype handling

 This may be a simple set of macros to set/unset flags

C++ Wrappers

It might be prudent to also create proper C++ wrappers around libcouchbase, so that one may be able to do something like:

LCB::Result& result = lcb->get("foo");


It has been recommended in the past that this be provided as an external library which will be called lcb_util. While the matter of whether this functionality be included inside an 'lcb_util' or integrated into libcouchbase itself is important, we will discuss several logistics, and pros/cons of both.

As a Standalone Library

Having an lcb_util would allow a different branch of development to proceed for somewhat higher level functions from which all SDKs can develop with a guarantee that this would not hinder and/or affect core protocol/network handling functionality.

The drawbacks involve the complexity of deployment (the goal is really to make most SDKs use these common utility functions which may be tested) - as such, it would now require the user to install two libraries instead of one - a detail which we may not wish to bother the user with.

Additionally, it would require modularization of libcouchbase's testing framework so that the API functions within this utility library can be tested. If the new functionality will be part of the SDKs, it will need to receive the same testing coverage and frequency as libcouchbase itself. This would require deploying libcouchbase's test functionality to an API level so that it may be used by the new utilities.

As Part of libcouchbase

Having this functionality included within libcouchbase would avoid the deployment issues mentioned above. It would also allow for better optimization and integration when modifications to libcouchbase itself are needed (rather than requiring modifications to the API itself).

Additionally, the complexities in testing are reduced as the functionality of these utility/extension features are tested whenever libcouchbase itself is tested.

The drawbacks involve slowdowns in modification and development, with changes to the utility methods requiring the same level of scrutiny and release frequency as libcouchbase itself.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.