Using spymemcached in a netty based project
I'd like to use spymemcached in a project based on netty. Netty's threading model uses 'boss' and 'worker' threads where none of the threads performs a wait for I/O or other external activity. The model used by spymemcached assumes that a thread uses a future to wait for completion of an operation. Internally spymemcached has a operation model with callbacks on activity completion.
In order to use these libraries together I see two alternatives:
1) Create a separate netty thread pool for spymemcached operations. This pool is independent of the main netty worker thread pool. Spymemcached operations are initiated and wait for completion using these threads.
2) Add support for listeners within spymemcached Callback classes that would be notified on operation completion. This allow passing results directly to the netty worker thread pool. This follows the netty threading model more closely and makes more efficient use of threads since spymemcached already has a worker thread that is performing I/O activity.
I would prefer to use option 2. Is option 2 something that spymemcached has considered? Do the developers have any thoughts on its viability?
I don't want to have to do a get() in the main thread on the future that would cause the main thread to wait for the completion of the memcache operation.
What I would like to do is add a listener to the GetOperationCallback().complete() that is invoked directly by the Memcache IO thread. In that listener I can kick one of the netty worker threads to process the data. The netty worker thread can do a future.get() knowing that the operation is complete and it won't have to wait for the results.
Tim,
I see what you mean, the idea to define callbacks on the operations to finish makes sense but is currently not implemented. Behind the scenes, we use callbacks for this stuff anyway, but there is no interface for this on the client side.
We can take this in consideration for a future release, but it's currently not possible. Thanks for bringing it up though!
As a workaround, you can pass the futures of to a separate thread and act on them there.
Thanks,
Michael
Hi Tim,
what kind of specific use case do you have in mind? All operations in the couchbase client (which uses spy underneath) are asynchronous, so its up to you how and when to wait for the operation result to come back.
I know that some people are using couchbase client together with netty and it seems to work for them. If you have a specific example in mind I may be help to guide you in the right direction!
Btw, if you are using Couchbase Server, please use the Client SDK from here: http://www.couchbase.com/develop/java/current. With it you can use all the features ouf Couchbase Server 2.0. (you can use spymemcached standalone against memcached-instances of course).
Thanks,
Michael