Moxi compatibility with other memcached clients

eran,

I am working on getting you expert help on this.

Thanks for your patience.

Bhawana

 Eran, there are a few concepts at work here, so please bear with me.

Membase employs the idea of vbuckets (documented here:http://wiki.membase.org/display/membase/vBuckets) in order to achieve our rebalancing, failover, etc.  This requires the client library to be "aware" of vbucket ids and other data in order to work properly against Membase directly (when accessing it over port 11210)

Now, to make sure that we are completely compatible with the standard memcached clients, we have developed a proxy (Moxi) to handle the translation between the standard protocol and the vbucket-aware protocol.  

When you access Membase on any port other than 11210, you are going through a Moxi process that lives by default on the server.  In this situation, the client's own hashing algorithm doesn't really matter at all since it is Moxi that makes the ultimate decision where a key lives.  While this allows for complete compatibility, it will also lead to an extra network hop when accessing some subset of your data.  To get around this, it is recommende to run that same Moxi process locally on your client.  Either way (server-side or client-side) should provide you the same access to the same keys.

What I "suspect" you were doing originally was accessing the Membase servers on port 11210 with a standard memcached client.  Unfortunately this is not supported, and will actually not work in most situations.  We are in the process right now of upgrading the spymemcached client to support vbuckets.  It is currently in beta and you can download the code here: http://wiki.membase.org/display/membase/prerelease+spymemcached+vBucket.  This client will allow you to access the Membase servers directly (through port 11210) without going through Moxi.  Even with using this client, if you then try to access the same data THROUGH Moxi (on port 11211), you should be able to access the same keys without trouble.

You are correct that our recommended best practice is to either use a smart client (the updated spymemcached) that can access the Membase servers directly, or to use a client-side Moxi.  I'm especially curious to hear about the benchmarking and performance overhead that you saw when using a client-side Moxi since it is designed to be a very lightweight process and has given us much success so far.

Depending on your setup, I think you have a few options:

1) Install two client-side Moxi processes.  One will point at your "local" cluster and the other will point at your "remote" cluster.  From your client, you will direct traffic at one or the other depending on the availability.  This should be a simple solution and I'll be happy to work with you to make sure the performance of the client-side Moxi is up to your standards as I'm sure we can get it there.  

2) Use the updated spymemcached client to connect to both clusters.  The client has persistent connections which should help reduce the overall number of connections you make to the remote cluster when necessary.  This should work fine today, but I can't yet recommend the client for production use as it is still in beta.

3) At the risk of adding even more confusion, you could do something even more complicated.  Use the updated spymemcached client to access your local cluster directly and setup a standalone-Moxi "tier" for accessing the remote cluster.  This would look like a few servers running the client-side Moxi that ALL of your clients point to in order to access the remote cluster.  This would give you the advantage of further reducing the number of connections to  your remote cluster (because you would only have the connections coming from your Moxi "tier" and not each client machine).

The bottom-line is that regardless of what (correct) method you use to access a Membase cluster, you should have access to all the data.

I hope I was able to explain everything clearly enough, please feel free to follow up if you need any further clarification.

Perry

Perry,

Thanks for your detailed answer.

What I was trying to acheive is exactly your 3rd option. In fact we have an existing setup that is just that, except that we're using a very old version of the Java memcached client (danga), and an old moxi version. Both client and moxi were hacked to make them use the same Ketama hashing algorithm, and hashing method (MD5). The problem is there was a bug somewhere in this setup, and we were trying to upgrade the clients / moxis to get over that.

Before upgrading, I wanted to test the client / moxi for compatibility. Since there's no compatibility I wated to mesure the moxi overhead. I tested this using the following setup:

• 4 memcached instances running on my desktop. This is a very 'standard' non production hardware.
• A client running on my laptop (Lenovo w510).
• I used the Xmemcached benchmark code to get the results.
• I executed the same benchmark once with the clients accessing the cluster directly, and once via the moxi.
• The moxi was running with no special configuration: just with "-d -z port=server1:port,server2:prot..."

The results I got were that the moxi served about 10 times less keys. I'm aware that these results may be affected by the setup, so I tried to run this with production machines. When I installed moxi on CentOS on prod mahcines, I got timeouts when connecting through it. Actually I got a stats response in a few seconds when connecting with telnet. I don't know why this happened, so I'll keep investigating and post my benchmark results when I get this working properly.

-Eran

 Thanks for your response Era.

When using Membase buckets, you must use the fixed hashing algorithm since it is directly tied to our use of vbuckets.  If you are just using memcached buckets it shouldn't matter...but I'm assuming you're using a Membase bucket here.

Can you test this setup out with the latest Membase code and latest Moxi code (un-hacked) to make sure that what I'm describing is what you're actually seeing? 

Looking at your example Moxi command-line, it looks like you're setting Moxi up to treat the Membase servers as regular memcached servers (the -z option...).  With the setup you described, you'd actually be going through TWO moxi hops (one on your local client, and another on the Membase server) so I would expect performance to be fairly poor.  With the latest code, Moxi is "smart" when talking to Membase servers and you should be running it like this:

"./moxi http://<IP of Membase Server>:8091/pools/default/bucketsStreaming/bucket_name" and replace the last "bucket_name" with the name of your bucket within the Membase cluster.  You can also look here for more instructions on how to configure Moxi to work against Membase: http://wiki.membase.org/display/membase/Standalone+Moxi+Component

Let me know if this works out better for you.  We might be able to setup a quick phone call to discuss further if you like.  Shoot me a private email if you're still having troubles.

Thanks much, take care.

Perry

Perry,

I'm not using membase. I'm indeed using memcached.

I performed some testing today, and will be posting the results in a min.

Hi Perry,

I've completed a cycle of performance testing. Here are the results:

Direct access from client to memcached cluster:
threads=1,repeats=40000,valueLength=64,tps=3323,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=512,tps=3046,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=1024,tps=2624,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=4096,tps=1735,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=16384,tps=970,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=10,repeats=40000,valueLength=64,tps=52482,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=512,tps=44345,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=1024,tps=35681,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=4096,tps=18279,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=16384,tps=6956,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=50,repeats=40000,valueLength=64,tps=141442,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=512,tps=116683,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=1024,tps=93925,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=4096,tps=28226,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=16384,tps=7139,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=100,repeats=40000,valueLength=64,tps=190994,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=512,tps=157004,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=1024,tps=106560,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=4096,tps=28365,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=16384,tps=7134,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=300,repeats=40000,valueLength=64,tps=250777,miss=0,fail=0,hit=9601200,all=12000000,hitRate=1.00
threads=300,repeats=40000,valueLength=512,tps=190506,miss=0,fail=0,hit=9601200,all=12000000,hitRate=1.00
threads=300,repeats=40000,valueLength=1024,tps=108057,miss=0,fail=0,hit=9601200,all=12000000,hitRate=1.00
threads=300,repeats=40000,valueLength=4096,tps=28339,miss=0,fail=0,hit=9601200,all=12000000,hitRate=1.00
threads=300,repeats=40000,valueLength=16384,tps=9810,miss=4399200,fail=0,hit=5202000,all=12000000,hitRate=0.54

Local Moxi: 

threads=1,repeats=40000,valueLength=64,tps=2716,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=512,tps=2498,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=1024,tps=2471,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=4096,tps=1611,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=16384,tps=811,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=10,repeats=40000,valueLength=64,tps=4118,miss=1113,fail=0,hit=318927,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=512,tps=4028,miss=6353,fail=0,hit=313687,all=400000,hitRate=0.98
threads=10,repeats=40000,valueLength=1024,tps=3897,miss=10702,fail=0,hit=309338,all=400000,hitRate=0.97
threads=10,repeats=40000,valueLength=4096,tps=2840,miss=3661,fail=0,hit=316379,all=400000,hitRate=0.99
threads=10,repeats=40000,valueLength=16384,tps=2263,miss=1885,fail=0,hit=318155,all=400000,hitRate=0.99
threads=50,repeats=40000,valueLength=64,tps=4797,miss=109900,fail=0,hit=1490300,all=2000000,hitRate=0.93
threads=50,repeats=40000,valueLength=512,tps=4072,miss=32984,fail=0,hit=1567216,all=2000000,hitRate=0.98
threads=50,repeats=40000,valueLength=1024,tps=3899,miss=34272,fail=0,hit=1565928,all=2000000,hitRate=0.98
threads=50,repeats=40000,valueLength=4096,tps=2892,miss=20784,fail=0,hit=1579416,all=2000000,hitRate=0.99
threads=50,repeats=40000,valueLength=16384,tps=2242,miss=10761,fail=0,hit=1589439,all=2000000,hitRate=0.99
threads=100,repeats=40000,valueLength=64,tps=5426,miss=486823,fail=0,hit=2713577,all=4000000,hitRate=0.85
threads=100,repeats=40000,valueLength=512,tps=4187,miss=89721,fail=0,hit=3110679,all=4000000,hitRate=0.97
threads=100,repeats=40000,valueLength=1024,tps=3864,miss=74364,fail=0,hit=3126036,all=4000000,hitRate=0.98
threads=100,repeats=40000,valueLength=4096,tps=2909,miss=47645,fail=0,hit=3152755,all=4000000,hitRate=0.99
threads=100,repeats=40000,valueLength=16384,tps=2235,miss=26519,fail=0,hit=3173881,all=4000000,hitRate=0.99
threads=300,repeats=40000,valueLength=64,tps=6605,miss=2780933,fail=0,hit=6820267,all=12000000,hitRate=0.71
threads=300,repeats=40000,valueLength=512,tps=4835,miss=1298994,fail=0,hit=8302206,all=12000000,hitRate=0.86
threads=300,repeats=40000,valueLength=1024,tps=3973,miss=320935,fail=0,hit=9280265,all=12000000,hitRate=0.97
threads=300,repeats=40000,valueLength=4096,tps=2951,miss=552981,fail=0,hit=9048219,all=12000000,hitRate=0.94
 

I have 2 problems with the results:

1. There's a significant penalty for using moxi. At least with this default setup.
2. How come I get cache misses? When I perform the same test with no moxi on the way, there is a 100% hit ratio...

Here's the benchmark configuration:

• I used the code from here: http://xmemcached.googlecode.com/svn/trunk/benchmark/ (slightly modified for my needs)
• I use Xmemcached client configured with binary protocol, and a connection pool size = 1
• XMemcachedClient is given the memcached servers: memcs1:11213 memcs1:11212 memcs2:11213 memcs2:11212 memcs3:11213 memcs3:11212 memcs4:11213 memcs4:11212 memcs5:11213 memcs5:11212 memcs6:11213 memcs6:11212
• moxi was started with "-d -z 11411=memcs1:11213,memcs1:11212,memcs2:11213,memcs2:11212,memcs3:11213,memcs3:11212,memcs4:11213,memcs4:11212,memcs5:11213,memcs5:11212,memcs6:11213,memcs6:11212"
• When testing against moxi the XMemcachedClient was given a memcached server localhost:11411 (which is the local moxi server)

Is there any way to optimize the results to make moxi performance closer to a standard Java client?

The cache misses when using the moxi are very odd. Can you give me a hint?

 Thanks Eran.  I'm sorry if I'm still not clear on this, but are you using the Membase Server software at all or just Moxi against standard memcached servers?

Perry

Perry,

I'm using just Moxi against a standard memcached servers cluster.

We basically need Moxi to reduce the number of connections between Data Centers.

Since the hashing alg is incompatible with the existing clients, I tried to always use a local Moxi instead of directly accessing the servers. If I can't get a decent performance, I will have to look for another solution. I'm also worried about those cache misses - I didn't get these when the cluster calls were not being proxied by moxi.

Thanks much for your support - well appreciated.

-Eran

 Ahh, now I'm understanding a bit better...sorry for the initial confusion, I thought you were trying to setup Moxi against a Membase server.

Are you using the latest release of Moxi?  It should be using the libketama hashing algorithm and that should be compatible with any other client also using the same algorithm, provided they have the same list of IP addresses in the same order.

The cache misses are likely being caused by a difference in hashing algorithm as I mentioned above.  If you client thinks that "key_a" should live on server_1 due to its hashing algorithm, and Moxi thinks that it should live on server_2 due to a different hashing algorithm, then Moxi will receive a miss when it tries to retrieve that key.  Looking back through your post, if you use the latest Moxi and the latest spymemcached client (which we wrote as well) then the default hashing algorithms should be the same and you should see no cache misses (again provided they both are given the same list of IP's in the same order).  Can you try with the latest versions?

As for the performance, I would really expect  a client-side Moxi to give you just about the same performance as going directly to the memcached servers.  Take a look at this wiki page for Moxi: http://wiki.membase.org/display/membase/Moxi+Statistics#MoxiStatistics-TimingHistograms  This may help you see where it's possibly slowing down.  Looking  a little deeper into your results, there is a huge difference in tps when the local moxi was encountering a miss...that leads me to wonder if something in the test is not quite right since in general, a miss should be even FASTER than a hit.  Is it possible that Moxi was actually timing out for some reason and that your test is seeing that as a miss when it was actually just slow?  The first 5 or so lines of your test output look pretty comparable.  There's a small difference but I don't think it's significant.  Also, usually a better metric to measure is the actual latency of each request, rather than the tps.  Those two should generally track together, but latency is more important to your application.    You might also want to take a look at memcachetest or brutis which can run continuous (and customizable) workloads while recording latency and any failures.

Let me know if this helps you out.

Perry

I can't post my reply for some reason

eran,

Sorry to hear that you could not  post. Can you please describe the symptoms? Did you get an error message?

bhawana

Started moxi with -Z time_stats=1 (this was after about a 1/3rd of the benchmark):
stats proxy timings   
STAT 11411:default:connect    0+100=55567  24.74% *********
STAT 11411:default:connect  100+100=136970 85.73% ************************
STAT 11411:default:connect  200+100=30952  99.52% *****
STAT 11411:default:connect  300+100=1027   99.97%
STAT 11411:default:connect  400+100=49     99.99%
STAT 11411:default:connect  500+100=8     100.00%
STAT 11411:default:connect  600+100=2     100.00%
STAT 11411:default:connect  700+100=0     100.00%
STAT 11411:default:connect  800+100=0     100.00%
STAT 11411:default:connect  900+100=2     100.00%
STAT 11411:default:connect 1000+100=0     100.00%
STAT 11411:default:connect 1100+100=0     100.00%
STAT 11411:default:connect 1200+100=0     100.00%
STAT 11411:default:connect 1300+100=1     100.00%
STAT 11411:default:connect 1400+100=0     100.00%
STAT 11411:default:reserved     0+100  =190995   3.16% *
STAT 11411:default:reserved   100+100  =2148091 38.69% ******************
STAT 11411:default:reserved   200+100  =2789086 84.82% ************************
STAT 11411:default:reserved   300+100  =668285  95.87% *****
STAT 11411:default:reserved   400+100  =225162  99.59% *
STAT 11411:default:reserved   500+100  =18373   99.90%
STAT 11411:default:reserved   600+100  =3189    99.95%
STAT 11411:default:reserved   700+100  =2062    99.98%
STAT 11411:default:reserved   800+100  =762    100.00%
STAT 11411:default:reserved   900+100  =144    100.00%
STAT 11411:default:reserved  1000+100  =38     100.00%
STAT 11411:default:reserved  1100+100  =26     100.00%
STAT 11411:default:reserved  1200+100  =3      100.00%
STAT 11411:default:reserved  1300+100  =1      100.00%
STAT 11411:default:reserved  1400+100  =2      100.00%
STAT 11411:default:reserved  1500+100  =0      100.00%
STAT 11411:default:reserved  1600+100  =1      100.00%
STAT 11411:default:reserved  1700+100  =0      100.00%
STAT 11411:default:reserved  1800+100  =1      100.00%
STAT 11411:default:reserved  1900+100  =0      100.00%
STAT 11411:default:reserved  2000+100  =0      100.00%
STAT 11411:default:reserved  2100+200  =0      100.00%
STAT 11411:default:reserved  2300+400  =1      100.00%
STAT 11411:default:reserved  2700+800  =0      100.00%
STAT 11411:default:reserved  3500+1600 =0      100.00%
STAT 11411:default:reserved  5100+3200 =0      100.00%
STAT 11411:default:reserved  8300+6400 =0      100.00%
STAT 11411:default:reserved 14700+12800=0      100.00%
STAT 11411:default:reserved 27500+25600=1      100.00%
STAT 11411:default:reserved 53100+51200=0      100.00%
END

I also tried to fiddle with moxi's startup options (threads / protocol / -R / etc), but none gave a significant improvement.

Please tell me I'm doing something wrong...

10X, Eran

 Eran, have you been able to diagnose any further why you are getting cache misses?  As I mentioned before, I suspect that is another symptom of whatever problem that is causing your performance drop.  You're getting relatively the same tps when not experiencing any misses.

What is your client returning for these "misses"? 

Perry

Hi Perry,

Repoting a DM I sent - I think it was lost due to the forum system - it thought I was a spammer...

I'm using the latest version of moxi - downloaded from http://www.membase.org/downloads - I got the 64 bit rpm: http://c2988032.ltd.cloudfiles.rackspacecloud.com/moxi-server_x86_64_1.6...

I performed some further analytics:

Executed memcache-top v0.6 against the cluster:

INSTANCE        USAGE    HIT %    CONN    TIME    EVICT/s GETS/s    SETS/s    READ/s    WRITE/s  
memcs1:11213        33.2%    0.0%    362    1.0ms    0.0    361    0    10.6K    372.2K  
memcs1:11212        34.1%    0.0%    362    0.9ms    0.0    349    0    10.3K    359.6K  
memcs2:11213        33.3%    0.0%    362    1.0ms    0.0    389    0    11.4K    400.7K  
memcs2:11212        35.3%    0.0%    362    0.9ms    0.0    325    0    9906    335.3K  
memcs3:11213        32.2%    0.0%    362    0.8ms    0.0    379    0    11.2K    390.4K  
memcs3:11212        34.5%    0.0%    362    0.8ms    0.0    350    0    10.4K    360.6K  
memcs4:11213        36.4%    0.0%    92    0.9ms    0.0    297    0    9075    305.8K  
memcs4:11212        34.2%    0.0%    92    0.7ms    0.0    358    0    10.6K    368.8K  
memcs5:11213        36.6%    0.0%    317    0.9ms    0.0    303    0    9227    312.0K  
memcs5:11212        36.4%    0.0%    317    0.7ms    0.0    294    0    8870    303.1K  
memcs6:11213        33.9%    0.0%    317    0.8ms    0.0    354    0    10.5K    365.1K  
memcs6:11212        35.1%    0.0%    317    0.6ms    0.0    328    0    9915    338.3K  
 
AVERAGE:        34.6%    0.0%    302    0.8ms    0.0    341    0    10.1K    351.0K  
 
TOTAL:        12.5GB/    36.0GB        3624    9.9ms    0.0    4088    0    120.8K    4.1M  

And against the moxi:

INSTANCE        USAGE    HIT %    CONN    TIME    EVICT/s GETS/s    SETS/s    READ/s    WRITE/s 
moxi2:11411        36.6%    0.0%    3612    2.8ms    0.0    4083    0    120.7K    4.1M
 
AVERAGE:        36.6%    0.0%    3612    2.8ms    0.0    4083    0    120.7K    4.1M
 
TOTAL:        1.1GB/    3.0GB        3612    2.8ms    0.0    4083    0    120.7K    4.1M 

This shows the x3 performance degredation is on the moxi side.

My post above also shows that moxi is misbehaving.

In the test I see nulls when there are cache misses.

I will try again today with spymemcached client. Maybe Xmemcached client is the cause.

10X, Eran

 Thanks for your continued investigation and data.  I'm not sure I fully understand what memcache-top is showing in terms of difference between the two setups.  For example, the gets per second, read/s and write/s are almost identical which I would take to mean that both setups are running almost the same.  Also, against the "cluster", it seems that the total time was actually longer? (9.9ms versus 2.8ms)

Can you confirm which (very specific) hashing algorithm xmemcached is using?

Thanks Eran...sorry about the spam stuff, I think we've gotten that resolved for you now.

Perry

Hi Perry,

Thanks for putting so much time into this.

I think the time is showing the total - well, it reads "Total", and it is meaningless for this column - it's just a sum. The AVG is what really matters in this case.

Xmemcached is using array hashing by default, but I don't think it should make a difference when there's only one server (the moxi)

I rerun the benchmark with Spymemcached client. The results for running against the cluster are similar to the Xmemcached client. The results of running against the Moxi server are below:

threads=1,repeats=40000,valueLength=64,tps=2982,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=512,tps=2555,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=1024,tps=2538,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=4096,tps=1492,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=1,repeats=40000,valueLength=16384,tps=888,miss=0,fail=0,hit=32004,all=40000,hitRate=1.00
threads=10,repeats=40000,valueLength=64,tps=4004,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=512,tps=3772,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=1024,tps=3627,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=4096,tps=2779,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=10,repeats=40000,valueLength=16384,tps=2233,miss=0,fail=0,hit=320040,all=400000,hitRate=1.00
threads=50,repeats=40000,valueLength=64,tps=4192,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=512,tps=3936,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=1024,tps=3754,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=4096,tps=2824,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=50,repeats=40000,valueLength=16384,tps=2241,miss=0,fail=0,hit=1600200,all=2000000,hitRate=1.00
threads=100,repeats=40000,valueLength=64,tps=4299,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=512,tps=4020,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=1024,tps=3772,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=4096,tps=2829,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00
threads=100,repeats=40000,valueLength=16384,tps=2244,miss=0,fail=0,hit=3200400,all=4000000,hitRate=1.00

As you can see - there are no misses, still there's a terrible performance issue.

Perry,

Is it possible that something in moxi is throttling the requests?

If so, what are the command line arguments that control the throttling?

10X, Eran

 Off the top of my head I don't know of any throttling being done by Moxi.  I'm going to engage with one of our engineers over here to see if he can see anything here...I've just about exhausted my extensive knowledge, maybe it's not that extensive after all.

I'll get back to you as soon as possible.

Thanks Eran.

Perry

Hi Eran,

Just a quick pointer on throttling & control parameters in Moxi -- a good place to start is: http://wiki.membase.org/display/membase/Follow+A+Request+Through+Moxi

Cheers,

Steve