Home | Forums | Couchbase | Couchbase Server 2.0

querying large resultset throws error

4 replies [Last post]
Mon, 04/02/2012 - 22:17
divyapt
Offline
Joined: 03/06/2012
Groups: None

Hi ,

I have 4 million documents stored in couchbase . And i have a query that should fetch 3.8 million documents.

But when the query is executed (ViewResponse result = client.query(view, query);) It throws the following exception :

java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:545)
at com.nsn.sai.cache.views.TutorialViews$3.run(TutorialViews.java:372)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.ViewFuture.get(ViewFuture.java:65)
at com.couchbase.client.internal.ViewFuture.get(ViewFuture.java:47)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)

Is there a limit on the documents that can be queried ?

Thanks & Regards,
Divya

Top
Mon, 04/02/2012 - 23:36
ingenthr
Offline
Joined: 03/16/2010
Groups:

There shouldn't be, no. You do need enough memory for the result, as Java does not (yet) have a streaming parser for view results. Ruby actually does.

I wouldn't expect a timeout exception though if you were to run out of memory, but it's possible. Does it correlate to GC activity?

Also, what version is this with. If it's not 2.0 DP4, then I'd highly recommend retrying there.

Top
Tue, 04/03/2012 - 00:27
divyapt
Offline
Joined: 03/06/2012
Groups: None

Oh k , so if i increase the heap size it should work right ?
Please correct me if i have understood it wrong.

I increased the heap size to (4096M)? The memory occupied by the data in couchbase(4 million docs in my case) is 645 mb. When i query 3.8 million docs , should it fit in the above heap space mentioned ?

But instead of increasing heap space can we query objects in batches , is there any interface for querying in batches ?

And yes I am using 2.0 DP4.

Thanks & Regards,
Divya

Top
Tue, 04/03/2012 - 00:49
ingenthr
Offline
Joined: 03/16/2010
Groups:

There is actually built in support for accessing the same query in paginated chunks. Have a look at the paginatedQuery method and the Paginator object it returns.

This would not require all of the items to be in memory. You should be able to move over them in pages.

Top
Tue, 04/03/2012 - 03:03
divyapt
Offline
Joined: 03/06/2012
Groups: None

Yes it works thank you . I missed out reading about paginators. If possible please include it in the Java client documentation. it would be helpful for everyone. I am not sure if this info is documented elsewhere . sorry if i have missed out reading about it somewhere.

Top