Home | Forums | SDKs | SDKs

Java SDK 1.1 - Asynchronous Implementation of operation requests

4 replies [Last post]
Wed, 01/09/2013 - 03:32
alemar
Offline
Joined: 01/09/2013
Groups: None

Hi everybody,

I run into the following errors after this scenario:

- 1 Couchbase Server 2.0
- 2 Buckets of type Couchbase (enabled to flush, no replicas)
- 1 Java Client Implementation for handling the buckets

Data:
- About 7.5 million records with each import.
- One import-process is running on one bucket while the other bucket is used for the live environment.
- Since there is no delta between new and old data, I have to flush the bucket every time before the import starts.

Implementation of the flush:

final CouchbaseConnectionFactory cf = cfMap.get("STAGE");
ClusterManager cm;
try {
	cm = new ClusterManager(nodeList, LOGIN_NAME, LOGIN_PW);
	final FlushResponse response = cm.flushBucket(cf.getBucketName());
	LOG.info("Flush Response status: " + response);
} catch (final Exception e) {
	LOG.warning("Cannot flush bucket '" + cf.getBucketName() + "': " + e);	
} finally {
	if(cm != null) { cm.shutdown(); }
}

Problem:
- Too much data, to get the FlushResponse it takes a while
- Exactly after 60 seconds (messured with a profiler) a RuntimeException with the HTTP Error 503 is thrown

Conclusion:
- Timeout while it is waiting for the webservice response from the couchbase server?

Questions/possible solution (but how to implement?):
- Asynchronous implementation of the method which is flushing the bucket (including a Callback Object or an Observer implementation).
- How will the client in the SDK ever find out what the current progress of the flush request is if its just waiting for the OK response but nothing else?

Other workarounds:
- Please let me know if you have any solution!!!

Thanks

Top
Wed, 01/09/2013 - 04:12
daschl
Offline
Joined: 05/19/2012
Groups: None

Currently, the way this is implemented is that there is no asynchronous way to do flushing properly. This is planned for a future version, but not available in 1.1.0.

What you can do as a workaround is just to log the exception and then check if there are docs in the bucket. I know this is kind of painful, but enhancements in this area are planned for sure. The reason why this is currently a little bit of in a transition is that we switched from memcache-flush over to a http-based one.

http://www.couchbase.com/issues/browse/JCBC-208

Thanks,
Michael

Top
Wed, 01/09/2013 - 04:20
alemar
Offline
Joined: 01/09/2013
Groups: None

Thanks for the fast response. Yeah, unfortunately it´s a harsh approach to check for documents.
But how to find out how many docs are in the bucket? With checking the number of documents and if its decreasing I could probably avoid endless loops of document checks.

Thanks in advance

Top
Wed, 01/09/2013 - 04:25
daschl
Offline
Joined: 05/19/2012
Groups: None

A quick way is to write a view with a _count reduce function.

Like this:
map:
function (doc, meta) {
emit(meta.id, null);
}
reduce:
_count

Top
Wed, 01/09/2013 - 08:22
alemar
Offline
Joined: 01/09/2013
Groups: None

Hi,

just tried it out. The View is working perfect, but I´m facing another problem:

While the Bucket is flushing (state: unhealthy), I cannot use the client to access the view, since the flushing Operation is "blocking" all incoming requests, what ends again with an Timeout Exception after 60 seconds (Also happens if you try to show the result of your view in the web-console while it´s still flushing).

Never mind it is blocking, but I won`t to hack an endless loop throwing Timeout Exceptions every 60 seconds until the job is done.

Thanks

Top