How to Import from file with Nested JSON

Couchbase Server
1

I have a data file that I would like to import into Couchbase. It has Nested JSON (subdocuments). Here is the first line of many similar lines, notice the custom_params which is nested JSON which may have multiple elements not just Level:

{“ts”:1499521698021,“userid”:“ae2dffe31b0da46b1bb2149df1247dbb”,“sessionid”:“1655264457640160833”,“remote_ip”:“66.41.**.",“platform”:“iPhonePlayer”,“sdk_ver”:“u5.6.0f3”,“debug_device”:false,“user_agent”:“OhSheep/14 CFNetwork/811.4.18 Darwin/16.5.0”,“submit_time”:1499521735577,“name”:“Wool Level”,“custom_params”:{“Level”:“Level 5”},“appid”:"78b2d999-2821-4b4a-9c1e-”,“type”:“custom”}

2

Any updates? I’m hoping to incorporate this process into a blog article

3

It’s been a week now, any updates? This is blocking my project.

4

Hi,

Would the following accomplish what you need?:

https://developer.couchbase.com/documentation/server/current/tools/cbimport.html

The cbimport tool allows importing of JSON data. If you need something a little more custom that that, you might take a look at a utility I personally wrote:

https://blog.couchbase.com/import-mongodb-collection-data-couchbase-server-golang/

It is essentially a Golang script that will read a file line by line and import it into Couchbase. The script I wrote was intended for importing MongoDB data so it does some custom stuff, but you can see how you might incorporate your own custom stuff through it.

Let me know if that helps.

Best,

5

Sorry, I should have just tried it first! cbimport successfully imported the file (along with the nested JSON) using the syntax:

cbimport json -c couchbase://127.0.0.1 -u Administrator -p password -b UnityIAP -d file://rde.json -f lines -g key::%ts%

From the example here:

https://developer.couchbase.com/documentation/server/current/tools/cbimport.html

Worked first time.

6

Couchbase rocks! I created this KB article for exporting Unity game play data into Couchbase https://support.unity3d.com/hc/en-us/articles/115004052703

1 Like
7

Hi…
I have a export json file having data like below:

{“indent”:0,“lineItemID”:1,“metaid”:"Profit:1",“smeFinalLanguage”:“General Statement”}
{“indent”:1,“lineItemID”:2,“metaid":"Profit:2”,“smeFinalLanguage”:"Financial statements "}

here “metaid” is the document unique id [retrieved from using meta().id] .

I want to import this using below command.

#cbimport json --cluster http://localhost:8091 --bucket sample --dataset file:///opt/couchbase/bin/data.json --format lines --username <> --password <> –generate-key ID:#MONO_INCR#

Can you please tell me how can I use the “metaid” filed in “–generate-key” field. So that, same meta().id can be imported in new environment. Actually I have a requirement to export and import the same data including meta().id from production bucket to stage bucket.

8

See my syntax above. I used the “ts” field from my import using key::%ts%

9

Thanks for your help.
I have run cbimport command using this "–generate-key %metaid% " and it works.
But, I don’t want this field within the document value . So , is there is any way to omit this attribute loading under document value.

10

Yes, use the syntax I’ve described. Unless key and generate-key are the same operations. Please show an example of your import data, and what you are expecting.

11

Ok, here is the example …

Export json file :
{“indent”:0,“lineItemID”:1,“metaid”:“Profit:1”,“smeFinalLanguage”:“General Statement”}
{“indent”:1,“lineItemID”:2,“metaid":"Profit:2”,“smeFinalLanguage”:"Financial statements "}

Now import this data like below :
ID Value
Profit:1 {“indent”:0,“lineItemID”:1,“smeFinalLanguage”:“General Statement”}
Profit:2 {“indent”:1,“lineItemID”:2,“smeFinalLanguage”:"Financial statements "}

Please help.

12

What happens when you use the syntax that I provided?

13

While importing the data using your syntax, it is showing like below:
ID Value
Profit:1 {“indent”:0,“lineItemID”:1,“smeFinalLanguage”:“General Statement”,“metaid”:“Profit:1”}
Profit:2 {“indent”:1,“lineItemID”:2,“smeFinalLanguage”:"Financial statements ",“metaid”:“Profit:2”}

used below command:
cbimport json --cluster http://localhost:8091 --bucket sample --dataset file://file.json --format lines --username Administrator --password password --generate-key %metaid%

14

That is not the syntax that I used, as I’ve mentioned. Even if the syntax is equivalent, simply remove the field after the import.

15

Ok… could it be possible to send the syntax / command for this cbimport as per the above requirement.

16

Just use the syntax I posted previously. I didn’t use generate-key, just key. As stated, if my syntax also doesn’t work, you could just delete the field after the import.