Couchbase Server

Synchronous writes seem to collapse the database

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: 2.1
  • Component/s: clients
  • Security Level: Public
  • Labels:
    None
  • Environment:
    4-node cluster, 32GB RAM per node, each machine with 4 SSDs.

Description

When performing updates against the database, we're able to get throughput of about 160,000 ops/second, with throughput to disk of about 21,000 ops/second (92% cache hit rate).

However, when we try to do either synchronous writes to disk or synchronous replication, the server fails. Either we get 100% error rates, or if we throttle input and add auto-retries, the throughput drops to tens of operations per second. Given the disk is happily doing over 20k ops/sec, we'd expect to be able to process traffic at that volume even with synchronous disk writes.

We're calling the update() method in the Java client with PersistTo.ONE (or ReplicateTo.ONE). This is done through YCSB.

Is this the correct approach? Is there something else we should be doing? What numbers should we be expecting?



Attached is the console screenshot. You'll see that a some of the keys are getting requests at a few thousand per second, but virtually all those requests are failing, causing a true throughput of close to zero.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Denis Nelubin added a comment -
We found that the replace() operation can fails if the observePoll() failed. And we see that it fails with ObservedTimeoutException even with PersistTo.ONE on small one-node cluster.
On four-node cluster, under low load, the writes completes successfully, but the overall rate is hundreds ops/sec (in cases when async operation can do more than 60k writes/sec).
On four-node cluster, under high load, the writes returns the errors, as we said before.

Could you comment while observe polling works so slow? Is this the experimental feature? When it should be used?
Show
Denis Nelubin added a comment - We found that the replace() operation can fails if the observePoll() failed. And we see that it fails with ObservedTimeoutException even with PersistTo.ONE on small one-node cluster. On four-node cluster, under low load, the writes completes successfully, but the overall rate is hundreds ops/sec (in cases when async operation can do more than 60k writes/sec). On four-node cluster, under high load, the writes returns the errors, as we said before. Could you comment while observe polling works so slow? Is this the experimental feature? When it should be used?
Hide
Permalink
Dipti Borkar added a comment -
can you take a look?
Show
Dipti Borkar added a comment - can you take a look?
Hide
Permalink
Mike Wiederhold added a comment -
Denis,

There is an observe polling interval that is configurable in the client. Setting it to a lower value will likely yield better results at the cost of more network traffic. Since the observe call doesn't block the connection we poll the server for persistence. If the observe poll time is too high it may cause some extra latency in the request. Also note that with the observe call you will need to factor in poll time, network latency, server persistence speed, and client sending and receiving time. With the async calls you just need to account for client send time so there is much more overhead to a synchronous call that must hit disk on the server.

Also, observe is not considered experimental but was designed with two use cases in mind. The first is to make sure new data is contained in a subsequent query. Since data needs to hit disk before it is indexed the observe command is useful in the situation. The second use case is for data that is considered to be very important and to which we want to define certain durability requirements.

Can you describe you use case here so I can get a better idea of whether or not observe is the correct feature to use?
Show
Mike Wiederhold added a comment - Denis, There is an observe polling interval that is configurable in the client. Setting it to a lower value will likely yield better results at the cost of more network traffic. Since the observe call doesn't block the connection we poll the server for persistence. If the observe poll time is too high it may cause some extra latency in the request. Also note that with the observe call you will need to factor in poll time, network latency, server persistence speed, and client sending and receiving time. With the async calls you just need to account for client send time so there is much more overhead to a synchronous call that must hit disk on the server. Also, observe is not considered experimental but was designed with two use cases in mind. The first is to make sure new data is contained in a subsequent query. Since data needs to hit disk before it is indexed the observe command is useful in the situation. The second use case is for data that is considered to be very important and to which we want to define certain durability requirements. Can you describe you use case here so I can get a better idea of whether or not observe is the correct feature to use?

People

  • Assignee:
    Dipti Borkar
    Reporter:
    bengber
Vote (0)
Watch (2)

Dates

  • Created:
    Updated: