Vew result is empty for very large document

Couchbase Server
1

I have installed couchbase server 3.0.3 on ubuntu 12.04.
I have 1 bucket and it has 1 document of size ~570 KB.
i have created view to retrieve certain information (few fields of json doc) from the document.
when i click on show result or query through API (java SDK), result is empty.

when i reduce document size to say 15 KB or reduce the number of fields in view o/p( thats what i tried), same veiw returns result.
Is there any limit on size of document that view can process ? As far as i know couch base supports document size upto 20 MB.

Is there any configuration parameter that we can set to process big document in view ?

[mapreduce_errors:error,2015-05-13T18:24:46.362,ns_1@192.168.56.102:<0.2963.0>:couch_set_view_updater:-do_maps/3-fun-0-:743]Bucket myex1, main group _design/dev_Test, error mapping document b26cd7ba-7e5a-43c9-9f59-ebbe2563ec2d:xyz:1 for view test: too much data emitted: 1048640 bytes

I tried setting few parameters in /opt/couchbase/etc/couchdb/default.ini, like

indexer_max_doc_size = 0 ; set to unlimited
max_kv_size_per_doc = 10485760000 ; some large value. tried by setting it to 0
as well.

but no luck.

Note: I am emitting 10 K records for a single doc. Can this be a problem (i think) ? any work-around?

2

Hi,

I’m sorry that this isn’t yet in our documentation.

The limits are set out in this bug report:

https://issues.couchbase.com/browse/DOC-40

However, the default max doc limit is 1MB, I believe, Usually we’d suggest splitting documents if you can as Couchbase works more efficiently that way.

@vmx are you able to share any insight here?

Thanks,

Matthew.

3

Thanks for the reply.
We need to have few billions(~10 billions) of items stored in DB as key value.
What is better :

  1. store them directly in couchbase bucket as key-value, Or
  2. store them in view index ? ( i tried to emit key-value through map , but facing above mentioned issue).

What is difference between storage structure of view index Vs bucket ?
I would like to know performance implications of each of them.

4

@matthew ,
Could you please comment on my reply?
Looking for your suggestion .

Thanks.

5

Hi,

I’m not sure I understand the question. Presumably you need to store them as key-value documents, whether you use a view index or not. So, perhaps I’ve misunderstood, in which case please correct me.

With that many documents, my suggestion is that you attempt to find a way to do as much pure key-value access as you can. That’s not to say you shouldn’t use views but I wouldn’t recommend them as your primary index.

If you describe your use case a little more, I’d be happy to talk in a bit more depth.

Storing a standard key-value pair in Couchbase is the least resource heavy method and gives you:

  • Strong consistency: read your own writes.
  • You make full use of the cache (view indexes are stored on disk).

So, using key-value gives you sub-millisecond response times for cached items.

Views run at a slight delay, depending on how long it takes to run the view that builds the index. So, in effect they’re eventually consistent. There’s also a CPU and RAM impact from running the view query.

I hope that helps.

Cheers,

Matthew.

6

Hi @kbaswaraj did you have any luck updating those settings? It looks like we are having the same problem as described in this topic. Thanks.