Slow drain performance on SSD drives
Hi all,
I'm evaluating Couchbase right now and I'm not very satisfied with the first results. I think I may be doing something wrong...
I'm using this box:
http://www.ovh.co.uk/dedicated_servers/eg_64g_ssd_max.xml
I setup a machine with Ubuntu Server 12.04 LTS and installed a fresh Couchbase 2.0 instance. Then, I created a default bucket with 48 GB of RAM. I've turned auto-compaction off.
Afer that, from another machine (same network, 1Gbps) I started the 'load' workload from this benchmark:
https://github.com/Altoros/YCSB
It simply inserts 100M documents. It is using 100 threads. I've limited the speed to 20k inserts per second. Documents are JSON-s with 10 fields, 100 bytes each - slightly over 1KB per document, about 100GB of raw data in total.
Everything worked fine, but I'm surprised with very low speed of persisting items to disk. It changes over time from 10K to 20K per second. Why not 100-200K?
I've lowered the values of mem_low_wat and mem_high_wat to 2GB and 4GB respectively. Do I assume correctly that this should trigger draining at full speed when memory usage goes over 4GB?
When the insertion process was finished I had about 10M of documents in the disk queue. Couchbase was slowly flushing it to disk...
Do I understand correctly that Couchbase is using couchdb for disk storage and couchdb is using append-only strategy? Assuming that I was expecting a bit higher performance...
During flushing the queue to disks, I've analyzed atop reports. The CPU usage was about 100-150% (800% is max as it 4-core node with HT) and the disk usage was about 20-30%: about 900-1000 reads per sec (20-30MB/s), and 80-100 writes per sec (also 20-30MB/s). All used by 'memcached' process.
Where's the bottleneck?
Why having a node with CPU idle in 80% and disks capable of writing more than 10x faster the performance of storing new items on disk was limited to 10-20k/sec?
I'm also a bit surprised with high number of reads... Couchbase should store all metadata in RAM - is that true? Why to read anything in such case?
I'll be very thankful if anyone can comment on this problem... Maybe I should tune the config a bit to unlock extra power? Or maybe (I hope not) Couchbase's design does not allow higher performance (too many locks, single thread or something like that)?
Thanks,
BR
Actually, the bucket was configured to use 1 replica, but there was only one node in the cluster. Could this affect the test results?
It was Couchbase 2.0 beta.
I've loaded over 100 GB of data. The bucket was configured to use 48 GB of RAM so it definitely cannot fit all data in memory.
An ideas? I can run some extra tests, reconfigure something, meassure any additional metrics if that would help...
I've upgraded to non-beta 2.0 release. It's a bit better, but still not as good as I expected. The drain perfomance is about 20-30k per second.
I've also done a basic read benchmark. I've used my already created database (100M docs, 1KB each, about 100GB in total). When I'm asking for random documents from a small set, so that all queries are served from cache, I've got really nice perfomance (70-80K QPS - I'm limited by a java client).
Then problem is when I start asking about completely random documents. The bucket is configured to use 48GB of RAM so the cache hit rate is about 40-50%. In that scenario I've got only 3.5k QPS.
During this test I've got following stats (on server): 50% CPU usage (800% is max - it's 4 core machine with HT), 30% disk usage, about 1-2K reads per second per drive (server has two SSD drives capable of doing 40K reads per second each).
As you can see it should handle a couple times higher load. Any ideas why I'm achiving such poor results?
On of my ideas was that Couchbase has some kind of limit for concurrent read operations. For spinning drives setting this to pretty low number may be a good idea, but to unleash the full power of SSDs you need to remove most of such limits.
I was looking in config files and documentation for any settings like this but I havn't found anything. Do such settings exist?
Any ideas what else can limit the perfomance in this case?
Do you know if the view index is updated only when the document is drained to disk? Not sure if index updating suffers from this same bottleneck.
Hi, not sure why exactly you're only getting 10-20k items/sec drain rate. But some questions and info...
Assuming it's no replicas configured, to simplify the math?
Also, is this with Couchbase 1.8.x or with 2.0 beta?
Both, btw, keep all metadata in RAM, but not necessarily values when it's under memory pressure. Not sure if you have evicted items in your test (non-dirty values evicted out of memory to make more memory space for other dirty items). A resident-item-ratio below 100% should be able to tell you that. If so, then the system can more easily become disk I/O bound when there's need to read or write stuff to disk.
Couchbase 2.0 does use append-only-btree (or copy-on-write-btree), but it also needs to do reads (for example, in order to read in btree nodes that need "modification"). Additionally, 2.0 performs occasional compaction of files, which would affect drain rates during the compactions, as that also contends for disk I/O.
But, even with all that, you should be getting higher drain rates, so still a mystery.