How to use curl to export/import all documents from/to a bucket as json (including id)?

[myuserid]

I am trying to use REST API to emulate the following CLI commands for individual bucket(s): cbexport json & cbimport json

NOTE: The buckets are on two different Couchbase clusters (i.e. in different AWS regions)

I can use the following REST API to return all the ‘rows’ in a bucket:

/pools/default/buckets/{bucket}/docs

And I can use the following REST API to return the contents of a specific document from the bucket:

/pools/default/buckets/{bucket}/docs/{id}

But I have not been able to find a REST API that will return all documents in a bucket (i.e. similar to cbexport json).

My next problem will be how to take that exported json and import them into the target bucket (again, via REST API).

Here is how I am running the cbexport:

cbexport json \
                --cluster couchbase://localhost \
                --threads {num-threads} \
                --username {user} \
                --password {password} \
                --output {file} \
                --no-ssl-verify \
                --format list \
                --bucket {bucket} \
                --include-key id

I tried looking at the following REST documents, but did not find what I was looking for:

https://docs.couchbase.com/server/current/rest-api/rest-buckets-summary.html
https://docs.couchbase.com/server/current/rest-api/rest-bucket-intro.html

If someone could point me in the right direction, that would be great!

Thank you.

[mreiche]

the query api is the right direction…

curl -v http://10.143.210.101:8093/query/service -u Administrator:password -d ‘statement=SELECT * FROM my_bucket’

[myuserid]

Thanks for the reply…but when I try to run the query as such:

curl -k --silent -u ${COUCHBASE_ADMIN_USERNAME}:${COUCHBASE_ADMIN_PASSWORD} "https://${COUCHBASE_HOST_NAME}:${COUCHBASE_API_HTTPS_PORT}/query/service" -d 'statement=SELECT * FROM ${BUCKET_NAME}'

I get the following error:

"code": 4000,
No index available on keyspace {bucket-name} that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online.

[mreiche]

Follow the instructions in the message.

[myuserid]

Yes, I saw that…but I did not have to create any indexes when I use the cbexport json tool. So, I was trying to have the same behavior … without this extra requirement. I am not in control over the administration of this bucket.

[mreiche]

If you want the same behavior as cbexport, why not use … cbexport?

If you want to use curl instead of cbexport, then you have to use the mechanisms available by curl/http.

[myuserid]

Thanks again for your quick reply and interest in assisting me.

@mreiche

If you want the same behavior as cbexport, why not use … cbexport?

Because I want to give this functionality to my customer in such a way that they do not have to login to remote couchbase server. The nice thing about the REST API is that it can be executed remotely using standard, simple curl.

I was hoping there was an equivalent REST API for cbexport & cbimport and I was just unable to find it…but, apparently, there is no such equivalent REST API.

I understand your point about using /query/service to perform these activities; however, taking the output of the SELECT * from bucket and feeding it into a formatted bulk INSERT statement (with separate VALUES for each doc, etc.) is not going to be very easy in my opinion.

On the other hand, the json output from cbexport can be directly used as input to cbimport … which is very clean…and is more what I am looking to achieve.

If you are saying that there is no such ‘cbexport/cbimport’ REST API, then that is a fair answer. It’s not exactly what I wanted to hear, but so be it.

[mreiche]

that they do not have to login to remote couchbase server.

The REST APIs also require logins.

SELECT * from bucket and feeding it into a formatted bulk INSERT statement (with separate VALUES for each doc, etc.) is not going to be very easy in my opinion.

The SELECT is trivial. And jq can be used to construct the INSERT for every document (or that could even be done by the SELECT).

[myuserid]

@mreiche

The REST APIs also require logins.

I meant logging in to the remote unix host which requires additional privileges.

@mreiche

The SELECT is trivial. And jq can be used to construct the INSERT for every document (or that could even be done by the SELECT).

Okay. Thank you for this. I am unfamiliar how to do this and I guess I will have to find out more information about this…although, again, it will require modifying the couchbase bucket itself to add the index which I am not sure if I will be permitted to do.

Thanks again for your help.

[mreiche]

I meant logging in to the remote unix host which requires additional privileges.

cbexport/cbimport do not require logging in to the remote host. The --cluster argument is a url. Just like curl takes a url.

cbimport json | Couchbase Docs

[myuserid]

@mreiche

cbexport/cbimport do not require logging in to the remote host. The --cluster argument is a url. Just like curl takes a url.

Yes…but then you need to have the cbexport/cbimport binaries installed…whereas ‘curl’ is universal.

[mreiche]

but then you need to have the cbexport/cbimport binaries installed

Yes. Just like if you provided a utility that used curl. It would need to be installed

Why not expose a REST service that calls cbimport/cbexport? Then users could invoke it using curl.

[myuserid]

@mreiche

Why not expose a REST service that calls cbimport/cbexport? Then users could invoke it using curl.

Exactly! This is what I am looking for…but I don’t want to write such a thing…I was hoping it was already written and packaged with the other Couchbase REST APIs. But alas.

[mreiche]

I was hoping it was already written and packaged with the other Couchbase REST APIs. . But alas.

The REST application would be two lines :

int exportResult = Runtime.getRuntime().exec(cbexport_cmd_and_args).waitFor();
int importResult = Runtime.getRuntime().exec(cbimport_cmd_and_args).waitFor();

Or see Cross Data Center Replication (XDCR) | Couchbase Docs

[myuserid]

@mreiche

int exportResult = Runtime.getRuntime().exec(cbexport_cmd_and_args).waitFor();
int importResult = Runtime.getRuntime().exec(cbimport_cmd_and_args).waitFor();

As far as I know, cbexport & cbimport are not available for installation as a ‘tools’ package (according to the backlog item: Loading... ). So, this would require the implementation to be on the same machine as a full couchbase installation.

To clarify why I specifically asked “How to use curl to export/import all documents from/to a bucket as json (including id)?”, here are some restrictions I am facing:

• I do not have ssh login access to couchbase server (and associated tools/binaries)
• I am not allowed to modify the buckets or make other administrative changes (like XDCR)
• I am not allowed to install any binaries that would require private licensing or additional cost

Based on my research, the existing bucket REST API does not support exporting/importing documents from/to a bucket as json (given the above restrictions).

I am now trying to see if I might be able to use the python client libraries to do the job of the thin-client I am looking for: Cluster object — Couchbase Python Client Library 3.0.6 documentation

[mreiche]

I do not have ssh login access to couchbase server (and associated tools/binaries)

Not required for cbimport/export

I am not allowed to modify the buckets or make other administrative changes (like XDCR)

It’s not needed for cbimport/export. But you should explain that your company is already paying for the functionality that is being requested.

I am not allowed to install any binaries that would require private licensing or additional cost

Just install CE to use cbimport/cbexport. It’s free.

[myuserid]

Thanks for directing me to the Community Edition (CE) of Couchbase and the fact that it is free and contains the cbexport/cbimport utilities. Although this is not quite the solution I was originally looking for, it has gotten me to a place where I can do something similar.

I have taken your suggestion and have chosen to run cbexport/cbimport via the CE docker image. This way, it does not require me to install the CE binaries and I have full functionality of the cbexport/cbimport from my remote client.

For others who are interested, here is a sample of the docker command I can run to export all docs from my bucket as a json file:

touch ./foobar.json
docker run \
	-v $(pwd)/foobar.json:/tmp/foobar.json \
	--entrypoint /opt/couchbase/bin/cbexport \
	couchbase:community-6.6.0  \
	json \
       --cluster couchbase://${COUCHBASE_HOST} \
       --threads 2 \
       --username ${COUCHBASE_USER} \
       --password ${COUCHBASE_PWD} \
       --output /tmp/foobar.json \
       --no-ssl-verify \
       --format list \
       --bucket foobar \
       --include-key id

Thanks again for your patience and creative thoughts on my post.