Home | Forums | Couchbase | Couchbase Server 2.0

memory error while loading a json file

3 replies [Last post]
Fri, 02/01/2013 - 08:47
abhish_gl
Offline
Joined: 01/31/2013
Groups: None

I am trying to load a json file using 'cbdocloader' utility.

json file size: 62gb
space available on data couchbase directory=360gb

The couchbase server is a 64 bit linux machine with 8gb ram.

command used:
./cbdocloader -n localhost:8091 -u Administrator -p password -b staging /tmp/to_load/

error stack trace:
----------------------------------------
[2013-02-01 10:33:12,809] - [rest_client] [139670977554176] - INFO - found bucket staging
Traceback (most recent call last):
File "/opt/couchbase/lib/python/cbdocloader", line 237, in
main()
File "/opt/couchbase/lib/python/cbdocloader", line 229, in main
docloader.populate_docs()
File "/opt/couchbase/lib/python/cbdocloader", line 189, in populate_docs
self.enumerate_and_save()
File "/opt/couchbase/lib/python/cbdocloader", line 151, in enumerate_and_save
self.save_doc(dockey, fp)
File "/opt/couchbase/lib/python/cbdocloader", line 125, in save_doc
raw_data = datafile.read()
MemoryError
-----------------------------------------------------------

Can someone provide more info on what I am doing wrong or is there a limit on file size?

Top
Fri, 02/01/2013 - 10:32
anil
Offline
Joined: 01/31/2013
Groups: None

I suspect it’s the format of the file, I would recommend to use simple python script to split each json object into multiple files to produce one json object per file. Then load the data into Couchbase using the cbdocloader tool. You can check this blog post (http://blog.couchbase.com/loading-json-data-couchbase) some examples and code (https://github.com/couchbase/couchbase-examples).

Hope that helps…

Anil

Top
Fri, 02/01/2013 - 11:28
abhish_gl
Offline
Joined: 01/31/2013
Groups: None

Thanks for your reply.

Its a 62gb file, so what chunks would you suggest to split to?

If I split it to individual json objects it will have few hundred thousand or millions of files? is that what you mean.

-Abhishek

Top
Sun, 02/03/2013 - 03:10
tgrall
Offline
Joined: 09/05/2012
Groups: None

Hello,

Inside Couchbase 2.0, the maximum size of a document associated to a key is 20Mb.

As you can see here:
https://github.com/couchbase/couchbase-examples/tree/master/gamesim-samp...
we use smal JSON document and each filename will be used as a key into Couchbase.

Can you do the same, each "document" from your application point of view should be associated to a key. So you need to split your document for this.

Regards

__________________

Tug
@tgrall

Top