Read performance

This is the code i am using

public CouchBaseClient() throws IOException, URISyntaxException {
memcachedClient = new MemcachedClient(Arrays.asList(new URI("http://test-cookie-master.np.wc1.yellowpages.com:8091/pools")), "default", "default", "");
}

public void callCouchBase(int arg) throws IOException {
long ptime = 0;
long gtime = 0;
for (int i = 0; i < arg; i++) {
final ProfileBuffers.ProfilePayload.Builder builder = ProfileBuffers.ProfilePayload.newBuilder();
buildValue(builder, i);
final String key = "xxx_subscriber@1828383838" + System.currentTimeMillis() + i;
final long putStart = System.currentTimeMillis();
memcachedClient.set(key, 0, builder.build().toByteArray());
ptime += (System.currentTimeMillis() - putStart);
final long getStart = System.currentTimeMillis();
final byte[] o = (byte[]) memcachedClient.get(key);
gtime += (System.currentTimeMillis() - getStart);
System.out.println(gtime);
final ProfileBuffers.ProfilePayload profilePayload = ProfileBuffers.ProfilePayload.parseFrom(o);
if (profilePayload.getPayloadCount() != 10) System.out.println("error in data");
}
System.out.println(ptime + " for " + arg + " writes");
System.out.println(gtime + " for " + arg + " reads ");
}

First off I wouldn't quit say you read performance is bad since your seeing get request take on average 500 microseconds. The reason your sets seem faster though is because you are using an asynchronous api call for sets and a synchronous api call for gets. What you want to do for a set call is this:

memcachedClient.set(key, 0, builder.build().toByteArray()).get();

This will cause the application thread to block until you get the result from the server. The reason we have the asynchronous API is so you can make a call to the server, do some other stuff, and then call get() on the Future when you actually need the value.

I also recommend using System.nanoTime() since most of our API call will happen in les than a millisecond.

Let me know if you have any other questions.

Thanks for the reply was just about to give up.

I switched to using couchbase 2.0 and did the changes that you suggested. After the switch to 2.0 and the changes
this is what I see

44 secs for 100000 writes
41 secs for 100000 reads

432 secs for 1000000 writes
405 secs for 1000000 reads

I get about 600 reads/writes per second per server. is this normal? I was expecting to see more , I am using tokyocabinet with the same data and get more that 5000 writes per sec.

with the 2.8-preview3.jar I am using

memcachedClient = new MemcachedClient(AddrUtil.getAddresses("np101.wc1.yellowpages.com:11211 np103.wc1.yellowpages.com:11211"));

will this slow the client cause it goes through moxi?

Moxi will definitely slow things down. Use the CouchbaseClient in the 2.8 developer preview and that should improve performance. Also, I would need more information on what you cluster looks like, how many threads (CouchbaseClient object) you are using, and what the data size is in order to give you a more accurate depiction of what you should expect.

I can tell you right now that I have been able to do well over 100k ops/sec for gets and close to that same number for sets (but I can't remember off the top of my head. I also want to mention that even though I have seen over 100k ops/sec for sets this throughput rate is not sustainable because at some point Couchbase is limited by how fast things can be written to disk.

I have cluster with two servers (8 core / 8 GB) and have allocated 6 GB of ram per each server. I am using a protocol buffer object as value about 450 bytes and key is around 20 bytes. I am already using the 2.8-preview3 client (still confused about which port is moxi port and which is not). I am using a singleton client from a simple test program
running a simple loop that does both the read an write , so its just one single thread that does it. I first want to test out how well just this setup performs, there are some scenarios we have in which we cannot run this with multiple threads.

np101 and np103 are my cluster machines and I am running the client from np102 (0.021 ms ping). I feel it would be helpful if there is a bulkSet in the api which takes in a map of keys and values as it would help to cut out on the network time.

memcachedClient = new MemcachedClient(AddrUtil.getAddresses("np101.wc1.com:11211 np103.wc1.com:11211"));

Thanks,
Suresh

>> (still confused about which port is moxi port and which is not)

You don't need to worry about which ports to connect to. Just use the constructor that looks like this

List<URI> uris = new List<URI>();
uris.add(URI.create("http://address:8091/pools" );
CouchbaseClient client = new CouchbaseClient(uris, "default", "");

This constructor will make sure everything connects to the right ports.

>> there are some scenarios we have in which we cannot run this with multiple threads

Your application may not benefit from having multiple threads, but you might have multiple application servers running your application. If you had 4 application servers for example, then you would have 4 threads running against your Couchbase cluster even though they are on different machines. My point here is that if you want to see what the performance of Couchbase is like you will want to be able to hit it with as much traffic as possible in order to see what workload is sustainable.

If you find that you can only use one thread in you application and the you have only one application server then the bottleneck will be your application and not Couchbase.

>> I feel it would be helpful if there is a bulkSet in the api which takes in a map of keys and values as it would help to cut out on the network time.

Under the hood Spymemcached actually aggregates set and get operations together and sends them as a multi-set. Just write you code like you are doing individual set operations and Spymemcached will do this for you automatically.

In my case I just have one app server, lemme explain the scenario , every day I get a flat file with about 100 million records that I need to process and push it to the nosql db. Currently using tokyocabinet and the process finishes within an hour.

There are several front end application servers that do a read after the loading process finishes. So the first perf test that I need to do is how fast can I load up this file into couchbase.

There are other scenarios where we do read/writes simultaneously from several app server machines into the tokyocabinet now , the scenario you are describing is more like that for this case I need to check How fast Can i load the data in couchbase with just one server.

Thanks for the help
Suresh

>> every day I get a flat file with about 100 million records that I need to process and push it to the nosql db

What you want to do sounds kind of similar to how our Hadoop plugin works except instead of reading from hdfs your reading from your local file system. What I did was to split the file into parts so that I could work on different parts of the file in parallel. For example one thread did the first quarter in the file and the second thread did the second quarter and so on. This may or may not be possible. If not, you can still achieve high throughput by using the async API properly. The other thing I did was to send off a bunch of set operation and putting the Future that they return into a queue. Then after I sent off 10,000 or so I would go through the queue and check the return statuses on each of the set operations to make sure that it got into Couchbase properly. If it didn't then I just send it again and put it back in the queue.

With large bulk loads you will have failures during set operations due to temp_oom (temporarily out of memory). This means that your writing to Couchbase so quickly that the disk cannot keep up this the amount of items that you have put into Couchbase. It's not a big deal because the disk will eventually catch up, but it means that you need to slow down a bit. If you try again later (even 1ms later) then the operation will most likely succeed.

What you will probably see is everything being extremely fast and that at some point slow down as a result of how fast your disk can process items. I also want to mention that the amount of memory in your cluster will have a great effect on how quickly you can bulk load. For example if you entire flat file can fit in memory I could write a script that could load all of the data in less than 20 minutes, but that would be in ideal conditions. The workload your cluster is under will also affect how fast this load can take place. If this is on a production system that is under load I would recommend throttling the amount of items loaded from the flat file per second.

Let me know f yo still have issues getting at least greater than 5k ops/sec.

Thanks, I already started a 100 million randon load with the async api. Will keep you informed how it goes.

thanks,
Suresh

I tried pushing in 100 million records , I tried the synchronous write function as I wasn't concerned about time it took I just wanted the the DB to be loaded up

I have a setup as described in my earlier messages.About 80-90% of writes seem to be timing out. I think it uses the DEFAULT_OPERATION_TIMEOUT (2500) which seems to be reasonably high. I am writing to it in batches of 1000. 15 million is the amount of records I could push with about 10 attempts.

I understand that we are limited with speed of disk I/O here still 15 million seems to be too low a number to get so many timeouts. What is the recommend server configuration for a setup like this?

THanks,
Suresh

Hello.

Since this is related to read performance, let me use the same thread and ask the similar question..

I was using the memcached for awhile for production environment, and I wanted to try the membase EE.

Using the single server (no replication) with 6 gb of ram of dedicated machines as membase-server with 8 core, I am getting an average of 1.5k ops/second..

Seems this is awfully low.... I don't understand how you guys get 100k ops/sec on average..

My test is as simple as it can get..

Basically, storing a simple serializable object (less than 20bytes at the most), and repeatedly reading the same object..

Foo foo = (Foo) CacheManager.getInstance().get("app", "group", "All Products", 10, callback);

int size = 1000000;

for (int i = 0; i < size; i++) {
foo = (Foo) CacheManager.getInstance().get("app", "group", "All Products", 10, callback);

assertEquals("foo", foo.name);
}

This takes a long time and looking at membase monitoring, I get the max of 1.8k ops/sec and usually hovers around 1.5k ops/sec..

Is this expected behavior?

The ping from machine1 to machine2 is as below..

PING 172.16.112.231 (172.16.112.231) 56(84) bytes of data.
64 bytes from 172.16.112.231: icmp_seq=1 ttl=64 time=1.15 ms
64 bytes from 172.16.112.231: icmp_seq=2 ttl=64 time=0.751 ms
64 bytes from 172.16.112.231: icmp_seq=3 ttl=64 time=1.07 ms
64 bytes from 172.16.112.231: icmp_seq=4 ttl=64 time=0.807 ms
64 bytes from 172.16.112.231: icmp_seq=5 ttl=64 time=0.842 ms
64 bytes from 172.16.112.231: icmp_seq=6 ttl=64 time=0.596 ms
64 bytes from 172.16.112.231: icmp_seq=7 ttl=64 time=0.779 ms
^C

Just tested locally hitting itself.. Still hovers around 1.5k ops/sec..

maybe this is normal...?

Sorry for the delay in my response. Can you check to see what the reason is fro your sets failing. You can do this by calling getStatus().getMessage() on the future objects that are returned from async calls.

My guess here is that your application is only able to send 1.5k ops/sec and that the server itself is not saturated. 1.5k ops/sec is very low. Can you provide more example you your application code so I can try to figure out what is wrong?

I also want to note here that when we got 100k+ ops/sec that I had multiple clients that had multiple client thread running against Membase. If I recall correctly it was something like 5 clients running 8-16 threads. With one client I was able to get up to 30k-50k ops/sec depending on the size of my values. When I was limited by the value sizes though it was because I had maxed out my gigE network. These tests were run with synchronous operations too.

Single client with 3 threads, and doing a async get. The get method is as below..

MemcachedClient cache = getCache();
Future future = cache.asyncGet(key);
Object value = null;
try {
value = future.get(1, TimeUnit.SECONDS);

if (value == null) {
LOG.debug("Cache Miss for [" + key + "]. Loading externally with TTL of [" + DEFAULT_TTL + "]");

T loadedValue = callBack.loadFromExternal(key);
if (ttl < 1) {
ttl = DEFAULT_TTL;
}

cache.set(key, ttl, loadedValue);
value = loadedValue;
}
}
catch (TimeoutException e) {
future.cancel(false);

return callBack.loadFromExternal(key);
}

I'm curious what happens when you call getCache() what happens. Are you getting a MemcachedClient from a connection pool or are you creating a new MemcachedClient?

Also, another way we can test to see what is going on is to just try something simple and see what happens. I'm assuming your using the Spymemcached 2.6 or 2.7 series. If your on the 2.8 dev preview then change the MemcachedClient objects to MembaseClient objects. Run the code below and let me know what you see for performance. I saw 7k-8k ops/sec and this is with only one thread. This code mimics what you are trying to do.

  public static void main(String args[]) throws Exception {
    List<URI> uris = new LinkedList<URI>();
    uris.add(URI.create("http://<your-ip>:8091/pools"));
    MemcachedClient client = new MemcachedClient(uris, "default", "");
    if (!client.set("akey", 0, "avalue").get().booleanValue()) {
      System.err.println("Set failed");
    }
    for (int i = 0; i < 1000000; i++) {
      if (client.get("akey") == null) {
        System.err.println("Get failed");
      }
    }
    client.delete("akey");
    client.shutdown();
  }

getCache() simply returns from the pool, and I am on 2.7.3 of spy and 1.7.2 of membase.

Let me run your test and report the stats.

Using your test with single thread, I get around 1600 - 1800 ops/sec..

I'm assuming you on linux. We usually install into /opt/membase but if you install path is different then you might need to edit this command a little bit. Can you do a

/opt/membase/bin/mbstats localhost:11210 timings

This will allow you to see how fast Membase is processing your requests.

Yes I am, and when I am running the command, I get the stacktrace below.. I am using redhat 4.1.2 x64

/opt/membase/bin/mbstats localhost:11210 timings
Traceback (most recent call last):
File "/opt/membase/lib/python/mbstats", line 282, in ?
main()
File "/opt/membase/lib/python/mbstats", line 279, in main
c.execute()
File "/opt/membase/lib/python/clitool.py", line 43, in execute
f[0](mc, *args[2:], **opts.__dict__)
File "/opt/membase/lib/python/mbstats", line 37, in g
f(*args[:n])
File "/opt/membase/lib/python/mbstats", line 112, in stats_timings
stats = sorted([seg(*kv) for kv in mc.stats('timings').items()])
File "/opt/membase/lib/python/mc_bin_client.py", line 316, in stats
cmd, opaque, cas, klen, extralen, data = self._handleKeyedResponse(None)
File "/opt/membase/lib/python/mc_bin_client.py", line 89, in _handleKeyedResponse
cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg()
File "/opt/membase/lib/python/mc_bin_client.py", line 71, in _recvMsg
raise exceptions.EOFError("Got empty data (remote died?).")
EOFError: Got empty data (remote died?).

Are you using the default bucket?

no. I am using a new bucket.. Does this matter?

I guess the script is defaulting to 'default'?

I have modified the test to use the 'default' and run the timings.. Here is the result

/opt/membase/bin/mbstats localhost:11210 timings
disk_invalid_item_del (1 total)
1us - 2us : (100.00%) 1 ################################################################################################
get_cmd (6088 total)
0 - 1us : ( 0.05%) 3
1us - 2us : ( 65.16%) 3964 ############################################################
2us - 4us : ( 89.90%) 1506 #######################
4us - 8us : ( 95.53%) 343 #####
8us - 16us : ( 99.89%) 265 ####
16us - 32us : (100.00%) 7
set_vb_cmd (1024 total)
16us - 32us : ( 29.20%) 299 ###########################
32us - 64us : ( 96.09%) 685 ##############################################################
64us - 128us : ( 99.71%) 37 ###
512us - 1ms : ( 99.90%) 2
1ms - 2ms : (100.00%) 1

Getting worse..

Now, the ops/sec is around 300 with 10 threads.. and I am getting tons of these

Running com.deposco.caching.CacheTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2011-12-02 18:24:25.556 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/172.16.112.231:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-12-02 18:24:25.907 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@32dcb03b
2011-12-02 18:24:25.980 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/172.16.112.231:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-12-02 18:24:26.051 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@5a4bf53e
2011-12-02 18:24:26.057 INFO net.spy.memcached.auth.AuthThread: Authenticated to 172.16.112.231/172.16.112.231:11210
2011-12-02 18:24:26.057 INFO net.spy.memcached.auth.AuthThread: Authenticated to 172.16.112.231/172.16.112.231:11210
2011-12-02 18:24:26.117 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/172.16.112.231:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-12-02 18:24:26.171 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@41584807
2011-12-02 18:24:26.175 INFO net.spy.memcached.auth.AuthThread: Authenticated to 172.16.112.231/172.16.112.231:11210
2011-12-02 18:24:26.236 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/172.16.112.231:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-12-02 18:24:26.282 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@5d15126e
2011-12-02 18:24:26.300 INFO net.spy.memcached.auth.AuthThread: Authenticated to 172.16.112.231/172.16.112.231:11210
2011-12-02 18:25:56.812 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769826 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.813 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769832 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.815 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769834 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.816 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769841 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.817 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769849 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.817 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769854 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.818 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769855 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.819 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769857 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.820 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769866 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.821 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769868 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.826 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769872 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.827 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769873 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.827 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769874 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.828 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769885 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.829 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769886 Key: %2Fapp%2Fgroup%2FAll+Products
2011-12-02 18:25:56.830 INFO net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Removing cancelled operation: Cmd: 0 Opaque: 1769888 Key: %2Fapp%2Fgroup%2FAll+Products

Hello,

I am a new bee evaluating couchbase.

I was able to scale the reads, Well i could get nearly 40K reads/sec (with bulk read).
With single get i used to get around 600 ops/sec. but with getBulk it has improved a lot.

But i am facing a strange problem, when i run multiple threads to get the data i face "OperationTimeoutException" exception very often.

Or even if not multiple threads, if i run it for 4-5 times in sequence i see this exception

net.spy.memcached.OperationTimeoutException: Timeout waiting for bulkvalues
at net.spy.memcached.MemcachedClient.getBulk(MemcachedClient.java:1298)
at net.spy.memcached.MemcachedClient.getBulk(MemcachedClient.java:1330)
at net.spy.memcached.MemcachedClient.getBulk(MemcachedClient.java:1344)
at ReadThread.run(ParallelReads.java:157)
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: sai133/10.58.116.133:11210
at net.spy.memcached.internal.BulkGetFuture.get(BulkGetFuture.java:120)
at net.spy.memcached.internal.BulkGetFuture.get(BulkGetFuture.java:49)
at net.spy.memcached.MemcachedClient.getBulk(MemcachedClient.java:1291)
... 3 more

Any clue how to come over this...

Thanks in advance,

BR,
Rajesh