Slow drain performance on SSD drives
I'm evaluating Couchbase right now and I'm not very satisfied with the first results. I think I may be doing something wrong...
I'm using this box:
I setup a machine with Ubuntu Server 12.04 LTS and installed a fresh Couchbase 2.0 instance. Then, I created a default bucket with 48 GB of RAM. I've turned auto-compaction off.
Afer that, from another machine (same network, 1Gbps) I started the 'load' workload from this benchmark:
It simply inserts 100M documents. It is using 100 threads. I've limited the speed to 20k inserts per second. Documents are JSON-s with 10 fields, 100 bytes each - slightly over 1KB per document, about 100GB of raw data in total.
Everything worked fine, but I'm surprised with very low speed of persisting items to disk. It changes over time from 10K to 20K per second. Why not 100-200K?
I've lowered the values of mem_low_wat and mem_high_wat to 2GB and 4GB respectively. Do I assume correctly that this should trigger draining at full speed when memory usage goes over 4GB?
When the insertion process was finished I had about 10M of documents in the disk queue. Couchbase was slowly flushing it to disk...
Do I understand correctly that Couchbase is using couchdb for disk storage and couchdb is using append-only strategy? Assuming that I was expecting a bit higher performance...
During flushing the queue to disks, I've analyzed atop reports. The CPU usage was about 100-150% (800% is max as it 4-core node with HT) and the disk usage was about 20-30%: about 900-1000 reads per sec (20-30MB/s), and 80-100 writes per sec (also 20-30MB/s). All used by 'memcached' process.
Where's the bottleneck?
Why having a node with CPU idle in 80% and disks capable of writing more than 10x faster the performance of storing new items on disk was limited to 10-20k/sec?
I'm also a bit surprised with high number of reads... Couchbase should store all metadata in RAM - is that true? Why to read anything in such case?
I'll be very thankful if anyone can comment on this problem... Maybe I should tune the config a bit to unlock extra power? Or maybe (I hope not) Couchbase's design does not allow higher performance (too many locks, single thread or something like that)?