Batch updates to Elasticsearch
I'm working on a project where we got high insert/update rate (10K/s average) and I'd like to use the Elasticsearch transport plugin to benefit from the rich queying capabilities of Elastcisearch.
The issue is that, most of our workload is CAS updates on counters which are fields inside Couchbase documents. Reads are infrequent relative to writes and we could happily cope with few hours of delay between the update in Couchbase and Elasticsearch. So, what I'd wish to do is to only sync between couchbase and elasticsearch once in a few hours, and only have the last version of a document send to elasticsearch (instead of all updates) in order to reduce the load on the ES cluster.
Is it possible to do this with the elasticsearch transport ?
The short answer is, not it is not possible with the Couchbase Elasticsearch plugin. ( http://www.couchbase.com/couchbase-server/connectors/elasticsearch )
As you probably know the Couchbase Elasticsearch Transport is based on the Couchbase XDCR (Cross Data Center Replication) protocol that replicates all the data that are persisted to disk.