Is this high disk write normal?
I'm running couchbase 2.0 RC3
I have a couchbase bucket that has about 6000 items in it and the following disk queue stats
ACTIVE REPLICA PENDING TOTAL
131 items 102 items 0 items 233 items
25 fill rate 33 fill rate 0 fill rate 58 fill rate
11 drain rate 19 drain rate 0 drain rate 30 drain rate
46.7 average age 36.3 average age 0 average age 41.5 average age
Here is a vmstat
vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 2 230596 403148 19596 104880 0 0 1 3 1 3 4 4 45 47
0 1 230596 404712 19596 104848 0 0 0 504 11864 7409 4 6 39 51
0 16 230596 405204 19600 105020 0 0 0 948 10251 6895 2 3 38 57
0 8 230596 404784 19608 105340 0 0 0 2300 11249 7176 3 4 26 67
0 5 230596 405256 19612 105544 0 0 0 188 10282 6368 2 4 25 69
iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 1589.65 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
6019 be/4 couchbas 0.00 B/s 78.70 K/s 0.00 % 97.84 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6012 be/4 couchbas 0.00 B/s 62.96 K/s 0.00 % 95.49 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6009 be/4 couchbas 0.00 B/s 39.35 K/s 0.00 % 89.98 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6011 be/4 couchbas 0.00 B/s 31.48 K/s 0.00 % 86.74 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6022 be/4 couchbas 0.00 B/s 62.96 K/s 0.00 % 86.50 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6013 be/4 couchbas 0.00 B/s 47.22 K/s 0.00 % 85.50 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6014 be/4 couchbas 0.00 B/s 70.83 K/s 0.00 % 82.71 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6008 be/4 couchbas 0.00 B/s 55.09 K/s 0.00 % 81.78 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6010 be/4 couchbas 0.00 B/s 47.22 K/s 0.00 % 81.27 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6016 be/4 couchbas 0.00 B/s 106.24 K/s 0.00 % 80.27 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6007 be/4 couchbas 0.00 B/s 47.22 K/s 0.00 % 74.92 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
1009 be/4 root 0.00 B/s 192.80 K/s 0.00 % 73.04 % [kjournald]
6018 be/4 couchbas 0.00 B/s 15.74 K/s 0.00 % 70.97 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6017 be/4 couchbas 0.00 B/s 55.09 K/s 0.00 % 68.62 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6020 be/4 couchbas 0.00 B/s 7.87 K/s 0.00 % 61.75 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6021 be/4 couchbas 0.00 B/s 23.61 K/s 0.00 % 61.03 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
6015 be/4 couchbas 0.00 B/s 31.48 K/s 0.00 % 47.67 % beam.smp -A 16 -sbt u -P~/couchbase-server.cookie"
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
This is a 3 node cluster with 1 replica per bucket.
Response time for my app is fine with no noticeable delays. Presumably because everything is being served from memory, but this seems like very high write usage for such a small set of data. I'm getting warnings every hour from my hosting provider (linode) with 10k blocks per second average. This also slows down the machine in other ways.
Should it be that high? As my load grows is that going to grow linearly as well, or does it just stay at the highest write rate possible?
- Brian
The docs are small but very frequently updated. Mainly uses to store analytics information for a SaaS service. So, the document(not sure if it even is a document in this case) would simply be an integer value and something like this:
accountID::month_day_year -> 123
so storing page views for a given day and given account.
Again, these do update frequently as one account might have a bunch of page views a second.
UPDATE: If I remove the 3rd server things look much better, as soon as I add a 3rd server to the cluster I get that high write and io wait.
Hello. Couchbase will write out the updates as fast as it can, de-duplicating writes when possible. Can you give some more info on your workload? How big are your docs on average and how often are you updating them? Does the write workload typically look random, or does it tend to update the same docs?
The write rate is impacted by both the total size of the dataset and the distribution of docs being updated, with a cost roughly O(N logN) for random writes. But if writing to the same subset of docs, many writes will be de-duplicated as the write queue ages, so cost growth can be sub-linear depending on workload.
Also, we have significant write optimizations coming that will both reduce CPU consumption and disk IO. But that might not reduce your total disk IO, only the write queue age. Again it depends on workload.