Skip to end of metadata
Go to start of metadata

This document describes the REST API exposed to access Couchbase

Open Questions
  • Expose documents and views APIs through the same HTTP socket.
    • Let people operate on two endpoints separately and optionally map them to the endpoints they want. It will be the most natural way with nginx, where everything built around location entity.
    • Develop some standard schema prefixes like _doc/ and _design, to allow to reference them, but here is another question "do we need to inject prefix in all places, like view results?"
  • Represent document keys so that they could be passed in URI. Urlencoding is quite common, but in views the server is using base64
  • Unlike memcache protocol, it should also encode bodies of the documents somehow, therefore MIME support. How we should do it?
  • low priority Evaluate what we need to support CouchApp-like deployment:
    • authentication and authorization!
    • proxy to syncgateway instead to raw couchbase?
    • new API to store attachments somewhere
    • additional useful APIs, like lists (can be done with multi-get)

There are several terms we are using while describing this API.

Term Definition
mount point The part of URI which where couchbase nginx module connected
document Arbitrary JSON or BLOB value. It could be referenced by its ID

Configuration and URIs

The Couchbase API could be mount to any endpoint in your application, and during configuration the bucket name and credentials should be specified like this:

server {
    location /wonderland/ {
        couchbase_pass example.com:8091,example.org bucket=wonderland;
    }
    ...
}

The config section above tell nginx to use /wonderland/ mount point to expose Couchbase API.

By default only Document API will be mounted, i.e. only CRUD operations can be executed there. Because in Couchbase itself both Document and Views APIs lives in completely different namespaces, the nginx module also keeps them separate. The example below will mount views and the documents into the common root.

server {
    location /wonderland/docs/ {
        couchbase_pass example.com:8091,example.org bucket=wonderland;
    }
    location /wonderland/views/ {
        couchbase_pass example.com:8091,example.org bucket=wonderland views=true;
    }
    ...
}

The connection configuration could be shared amound the locations using upstream

upstream couchbase {
    couchbase_server example.com:8091;
    couchbase_server example.org;
    couchbase_bucket wonderland;
}

server {
    location /wonderland/docs/ {
        couchbase_pass couchbase;
    }
    location /wonderland/views/ {
        couchbase_pass couchbase views=true;
    }
    ...
}

Document CRUD

The module maps basic data operations to HTTP methods. The table below assumes that the bucket mount to /storage location:

Couchbase operation HTTP method Comments
get "foo" GET /storage/foo  
add "foo" "bar" POST /storage/foo document goes in request's body, fails if ID exists already
  POST /storage the same as above, but will generate ID
set "foo" "bar" PUT /storage/foo document goes in request's body
delete "foo" DELETE /storage/foo  

Apart from HTTP methods, it is possible to use arguments in query string, to specify the command. To do so, the name of the parameter must be specified in the config:

server {
    location /wonderland/ {
        couchbase_pass example.com:8091,example.org bucket=wonderland;
        set $couchbase_cmd $arg_cmd
        set $couchbase_exptime $arg_exptime;
    }
    ...
}

The snippet above will translate this URI /storage/foo?cmd=replace&exptime=40 to the replace "foo" ... command with expiration of 40 seconds. In the same way can be controller ID, value, flags and expiration time.

The rest of the section demonstrates various example of usage the API, including optimistic locking with CAS. It assumes the following config:

server {
    location /wonderland/ {
        couchbase_pass example.com:8091,example.org bucket=wonderland;
        add_header ETag $couchbase_cas;
    }
    ...
}

Create document in the bucket

POST /wonderland
> Content-Type: application/json
{"name":"Alice","species":"human"}
< 201
< ETag: 10952782243130507264
< Location: http://couchbasenginx.apiary.io/wonderland/0a4683e4b4b161f505d0c2ae0c3a1abf

You can also specify ID with POST request if you need to create the document with given ID.

POST /wonderland/alice
> Content-Type: application/json
{"name":"Alice","species":"human"}
< 201
< ETag: 10952782243130507265
< Location: http://couchbasenginx.apiary.io/wonderland/alice

Note that when you are using POST request, the server is always supposed to create the new document. This means that you will get an error if the document already exists.

POST /wonderland/foobar
> Content-Type: application/json
{"foo":"bar"}
< 409
< Content-Type: application/json
< Content-Length: 74
{"error":"key_eexists","reason":"Key exists (with a different ETag value)"}

Update the document

PUT /wonderland/white-rabbit
> Content-Type: application/json
{"name":"Write Rabbit","species":"rabbit"}
< 204
< ETag: 10952782243130507266
< Location: http://couchbasenginx.apiary.io/wonderland/white-rabbit

You can also specify If-Match header and the server will perform optimistic lock before updating. This operation might fail 412 Precondition Failed

PUT /wonderland/duchess
> Content-Type: application/json
> If-Match: "wrong_etag"
{"name":"Duchess","species":"human"}
< 412
< Content-Type: application/json
< Content-Length: 74
{"error":"key_eexists","reason":"Key exists (with a different ETag value)"}

If the document being updated doesn't exist, you will get 404 Not Found error

PUT /wonderland/sheep
{"name":"Sheep","species":"sheep"}
< 404
< Content-Type: application/json
< Content-Length: 45
{"error":"key_enoent","reason":"No such key"}

Removing document from the bucket

DELETE /wonderland/caterpillar
< 204
< ETag: 10952782243130507267

You can also specify If-Match header if you need to ensure you are removing know version of the document.

DELETE /wonderland/bill-the-lizard
> If-Match: "wrong_etag"
< 412
< Content-Type: application/json
< Content-Length: 74
{"error":"key_eexists","reason":"Key exists (with a different ETag value)"}

If the document being removed doesn't exist, you will get 404 Not Found error

DELETE /wonderland/jabberwock
< 404
< Content-Type: application/json
< Content-Length: 45
{"error":"key_enoent","reason":"No such key"}

Getting the document by ID.

GET /wonderland/hatter
< 200
< Content-Type: application/json
< Content-Length: 36
< ETag: 10952782243130507268
{"name":"Hatter","species":"human"}

In case when the given document ID isn't exist in the bucket you will get 404 Not Found

GET /wonderland/red-queen
< 404
< Content-Type: application/json
< Content-Length: 45
{"error":"key_enoent","reason":"No such key"}

Use HEAD request to skip the document body

HEAD /wonderland/gryphon
< 200
< Content-Type: application/json
< ETag: 10952782243130507269

Each operation is carry ETag, which internally is CAS value. The ETag value can be used in the conditional requests with If-Match, If-None-Match. See examples of If-Match above.

Check if the key has been modified (in this case the server responds that it wasn't modified):

HEAD /wonderland/dormouse
< If-None-Match: 10952782243130507269
< 304

Couchbase Views

This section shows more advanced server feature: View indexes

One of the features of the Couchbase is ability to build efficient indexes leveraging Map/Reduce. They are called Views and you can define them on Admin Console UI.

The module allows you to query your views proxying them to Couchbase. All arguments will be transparently passed to Couchbase and the result will be streamed back.

As mentioned above, the view endpoint should be mount separately

server {
    location /wonderland/views/ {
        couchbase_pass example.com:8091,example.org bucket=wonderland views=true;
    }
    ...
}

For example we have a view all defined in the design document characters. It is simple map which will just emit all known characters (without any reduce function):

function (doc, meta) {
  emit(meta.id);
}

The result will look like

GET /wonderland/views/_design/characters/_view/all
< 200
< Transfer-Encoding: chunked
< Content-Type: application/json
{"total_rows":20,"rows":[
{"id":"alice","key":"alice","value":null},
{"id":"bill-the-lizard","key":"bill-the-lizard","value":null},
{"id":"caterpillar","key":"caterpillar","value":null},
{"id":"cheshire-cat","key":"cheshire-cat","value":null},
{"id":"dodo","key":"dodo","value":null},
{"id":"dormouse","key":"dormouse","value":null},
{"id":"duchess","key":"duchess","value":null},
{"id":"duck","key":"duck","value":null},
{"id":"eaglet","key":"eaglet","value":null},
{"id":"gryphon","key":"gryphon","value":null},
{"id":"hatter","key":"hatter","value":null},
{"id":"king-of-hearts","key":"king-of-hearts","value":null},
{"id":"knave-of-hearts","key":"knave-of-hearts","value":null},
{"id":"lory","key":"lory","value":null},
{"id":"march-hare","key":"march-hare","value":null},
{"id":"mock-turtle","key":"mock-turtle","value":null},
{"id":"mouse","key":"mouse","value":null},
{"id":"pat","key":"pat","value":null},
{"id":"queen-of-hearts","key":"queen-of-hearts","value":null},
{"id":"white-rabbit","key":"white-rabbit","value":null}
]
}

You can pass the any of supported query parameters, like include_docs=true for example (note that rev field replaced with etag:

GET /wonderland/views/_design/characters/_view/all?include_docs=true
< 200
< Transfer-Encoding: chunked
< Content-Type: application/json
{"total_rows":20,"rows":[
  {"id":"alice","key":"alice","value":null,"doc":{"meta":{"id":"alice","expiration":0,"flags":0,"etag":88499348547108864},"json":{"name":"Alice","species":"human"}}},
  {"id":"bill-the-lizard","key":"bill-the-lizard","value":null,"doc":{"meta":{"id":"bill-the-lizard","expiration":0,"flags":0,"etag":12227673967682453504},"json":{"name":"BilltheLizard","species":"lizard"}}},
  {"id":"caterpillar","key":"caterpillar","value":null,"doc":{"meta":{"id":"caterpillar","expiration":0,"flags":0,"etag":7907600423173816320},"json":{"name":"Caterpillar","species":"caterpillar"}}},
  {"id":"cheshire-cat","key":"cheshire-cat","value":null,"doc":{"meta":{"id":"cheshire-cat","expiration":0,"flags":0,"etag":15887700786432507904},"json":{"name":"CheshireCat","species":"cat"}}},
  {"id":"dodo","key":"dodo","value":null,"doc":{"meta":{"id":"dodo","expiration":0,"flags":0,"etag":14764617828380246016},"json":{"name":"Dodo","species":"bird"}}},
  {"id":"dormouse","key":"dormouse","value":null,"doc":{"meta":{"id":"dormouse","expiration":0,"flags":0,"etag":4027757040775528448},"json":{"name":"Dormouse","species":"dormouse"}}},
  {"id":"duchess","key":"duchess","value":null,"doc":{"meta":{"id":"duchess","expiration":0,"flags":0,"etag":17543907245979402240},"json":{"name":"Duchess","species":"human"}}},
  {"id":"duck","key":"duck","value":null,"doc":{"meta":{"id":"duck","expiration":0,"flags":0,"etag":13461959332744724480},"json":{"name":"Duck","species":"bird"}}},
  {"id":"eaglet","key":"eaglet","value":null,"doc":{"meta":{"id":"eaglet","expiration":0,"flags":0,"etag":8572459711327174656},"json":{"name":"Eaglet","species":"bird"}}},
  {"id":"gryphon","key":"gryphon","value":null,"doc":{"meta":{"id":"gryphon","expiration":0,"flags":0,"etag":14715372901595414528},"json":{"name":"Gryphon","species":"gryphon"}}},
  {"id":"hatter","key":"hatter","value":null,"doc":{"meta":{"id":"hatter","expiration":0,"flags":0,"etag":3615690869010661376},"json":{"name":"Hatter","species":"human"}}},
  {"id":"king-of-hearts","key":"king-of-hearts","value":null,"doc":{"meta":{"id":"king-of-hearts","expiration":0,"flags":0,"etag":6164167507172196352},"json":{"name":"KingofHearts","species":"card"}}},
  {"id":"knave-of-hearts","key":"knave-of-hearts","value":null,"doc":{"meta":{"id":"knave-of-hearts","expiration":0,"flags":0,"etag":205344449230864384},"json":{"name":"KnaveofHearts","species":"card"}}},
  {"id":"lory","key":"lory","value":null,"doc":{"meta":{"id":"lory","expiration":0,"flags":0,"etag":13830705845886844928},"json":{"name":"Lory","species":"bird"}}},
  {"id":"march-hare","key":"march-hare","value":null,"doc":{"meta":{"id":"march-hare","expiration":0,"flags":0,"etag":14847959710254039040},"json":{"name":"MarchHare","species":"hare"}}},
  {"id":"mock-turtle","key":"mock-turtle","value":null,"doc":{"meta":{"id":"mock-turtle","expiration":0,"flags":0,"etag":211262020811554816},"json":{"name":"MockTurtle","species":"turtle"}}},
  {"id":"mouse","key":"mouse","value":null,"doc":{"meta":{"id":"mouse","expiration":0,"flags":0,"etag":7144275470706606080},"json":{"name":"Mouse","species":"mouse"}}},
  {"id":"pat","key":"pat","value":null,"doc":{"meta":{"id":"pat","expiration":0,"flags":0,"etag":89110677012152320},"json":{"name":"Pat","species":"pig"}}},
  {"id":"queen-of-hearts","key":"queen-of-hearts","value":null,"doc":{"meta":{"id":"queen-of-hearts","expiration":0,"flags":0,"etag":8409231712625688576},"json":{"name":"QueenofHearts","species":"card"}}},
  {"id":"white-rabbit","key":"white-rabbit","value":null,"doc":{"meta":{"id":"white-rabbit","expiration":0,"flags":0,"etag":14777043409285742592},"json":{"name":"WhiteRabbit","species":"rabbit"}}}
]
}

Misc

You can find executable spec here: http://docs.ngxcbm.apiary.io/

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Feb 26, 2013

    Matt Ingenthron says:

    Is CAS also valid for a DELETE? Also, it may be useful to say something about w...

    Is CAS also valid for a DELETE?

    Also, it may be useful to say something about whether or not results from Couchbase are cached. I think they are not, but I am not certain.

    1. Feb 26, 2013

      Sergey Avseyev says:

      The won't be cached, but does it worth to introduce, X-Couchbase-CAS and X-Couch...

      The won't be cached, but does it worth to introduce, X-Couchbase-CAS and X-Couchbase-TTL when we have HTTP abstractions like ETag (If-None-Match) and Expires. Probably we can find a way to reuse HTTP semantic?

    2. Feb 26, 2013

      Sergey Avseyev says:

      And yes, it is possible to specify CAS for delete command. And it will fail unle...

      And yes, it is possible to specify CAS for delete command. And it will fail unless it match to the current one

  2. Feb 26, 2013

    Sergey Avseyev says:

    Another note: I probably will "normalize" view output, because we anyway should ...

    Another note: I probably will "normalize" view output, because we anyway should intercept the include_docs=true parameter. I'm going to rename "CAS" to "etag" in meta field, and hide flags completely from the user

    1. Feb 26, 2013

      Matt Ingenthron says:

      I like the idea of allowing for include_docs, but perhaps in a later implementat...

      I like the idea of allowing for include_docs, but perhaps in a later implementation. We should probably just work out what our initial release criteria is.

  3. Feb 26, 2013

    Aleksey Kondratenko says:

    My team has some of the same challenges as part of document editing API. Here's ...

    My team has some of the same challenges as part of document editing API. Here's list of tricky things that has to be addressed:

    • dealing with arbitrary octet strings as keys
    • dealing with user-defined vbucket mapping. I think it implies that either header or query parameter or something specifying vbucket id has to be supported
    • dealing with separate namespaces of design docs and normal docs
    • dealing with separate namespaces of views and normal docs. I.e. view queries should not ever be confused with GETting docs and vice versa
    • dealing with _all_docs use case
    • dealing with non-json docs
    • dealing with doc's metadata (not just CAS, but flags, expiration etc)
    1. Feb 27, 2013

      Sergey Avseyev says:

      Thanks, Aleksey I will update spec with the answers

      Thanks, Aleksey I will update spec with the answers

    2. Feb 27, 2013

      Sergey Avseyev says:

      The short answers are (for this iteration at least) dealing with arbitrary oct...

      The short answers are (for this iteration at least)

      dealing with arbitrary octet strings as keys

      The key part in URI might be be urlencoded. I don't like idea with base64 with BIN_ prefix

      dealing with user-defined vbucket mapping. I think it implies that either header or query parameter or something specifying vbucket id has to be supported

      On this iteration I would rather skip this feature

      dealing with separate namespaces of design docs and normal docs

      dealing with separate namespaces of views and normal docs. I.e. view queries should not ever be confused with GETting docs and vice versa

      I will just forbid to create documents with _design/ prefix, It won't be possible to perform administrative task via nginx module, only data access, so you won't be able to create design document there.

      dealing with _all_docs use case

      Won't implement this feature

      dealing with non-json docs

      Currently I can use either flags or datatype field to mark key as json in case the use specify application/json content-type on store. and after that I can just render application/octet-stream for all other documents. There no other issues with storing blobs in couchbase. In views they are already base64encoded, but for retrieving data I can just stream the contents

      dealing with doc's metadata (not just CAS, but flags, expiration etc)

      flags won't be exposed at all with nginx module. CAS will be abstracted as ETag with all related semantic. I will rewrite view results to hide flags, expiration (currently) and make CAS (ETag) more accessible. i.e. render {"meta":{"etag":"xxx"}} always, for both include_docs false and true. And pull it to the upper level in json tree (see view examples above).

      For this iteration I probably won't implement TTL, but it probably can be mapped to HTTP headers too.

  4. Feb 27, 2013

    Aleksey Kondratenko says:

    Sergey, I'm okay with you doing arbitrary API with arbitrary requirement. Let m...

    Sergey, I'm okay with you doing arbitrary API with arbitrary requirement.

    Let me note however that my team not only has real requirement for HTTP CRUD API, but also long term plan to have a proper, complete and supported HTTP CRUD API implementation.

    So if you build something that can't be compatible with complete implementation, then we'll have 2 incompatible HTTP APIs. I think that's suboptimal. And that's why I believe we should be careful now to have all bases covered.

    Real problem is that design docs/views namespace is separate from docs namespace. And this is not reflected at all in your proposal.

    How about /<bucket/d/<doc-id> as namespace for CRUD ops ?

    1. Feb 28, 2013

      Matt Ingenthron says:

      Alk, we should sync up about those requirements at some point. In any case, I ...

      Alk, we should sync up about those requirements at some point.

      In any case, I agree with trying to socialize and come up with one implementation that covers all of the bases, but we may want to do that in stages.

      Toward that end, with Sergey's ideas here, we should come up with what the end looks like and then determine what the first stage may look like I think.

    2. Mar 05, 2013

      Sergey Avseyev says:

      How about /<bucket/d/<doc-id> as namespace for CRUD ops ? Maybe bette...

      How about /<bucket/d/<doc-id> as namespace for CRUD ops ?

      Maybe better

      /<bucket>/_doc/<doc-id>

      similar to

      /<bucket>/_design/<ddoc-id>

      1. Mar 05, 2013

        Aleksey Kondratenko says:

        +1 for _doc

        +1 for _doc

  5. Apr 16, 2013

    Aleksey Kondratenko says:

    Pinging. From brief look at couchbaselabs repository it appears implementation ...

    Pinging.

    From brief look at couchbaselabs repository it appears implementation proceeds for original API which means it'll be inherently incompatible with future implementation(s).

  6. Sep 30, 2013

    Aleksey Kondratenko says:

    Revisited update. Much better. Almost perfect! Here are few things I've spotted...

    Revisited update. Much better. Almost perfect!

    Here are few things I've spotted:

    • my understanding is that DELETE and If-Match are not part of rfc. It is explicitly allowed for PUT requests however. And our usage for DELETE looks logical enough. So probably not a problem.
    • I don't like PUT returning 404 if replaced item doesn't exist. It'll be slightly inconvenient imho. I propose allowing it by default. And supporting If-Match: * which as per RFC enables "only if it already exists" behavior.
    • I think it makes sense to define headers for all other metadata we have today in memcached protocol