beam.smp writing a lot
I have a 2 node cluster setup of developer preview 2 release candidate 3.
I have a couchbase bucket with what I think is not a lot of items (26,240) that I'm using to track pageview statistics of websites. The actual documents stored are just numbers.
Bucket Name Nodes Item Count Ops/sec Disk Fetches/sec RAM Usage/Quota Disk Usage
pageviews 2 26240 338 0 12.8MB / 800MB 95.5MB
As shown above there are about 338 ops/sec at this sample time. That doesn't feel like a lot to me. One of the servers is writing out 5.6 M/s shown below in iotop. It looks like beam.smp is doing a lot of that.
I just don't understand why it would need to write THAT much for this type of load, something feels wrong. Causes a lot of io wait.
Total DISK READ: 0.00 B/s | Total DISK WRITE: 5.65 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1009 be/4 root 0.00 B/s 0.00 B/s 0.00 % 3.12 % [kjournald]
2619 be/4 couchbas 0.00 B/s 106.29 K/s 0.00 % 0.35 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2615 be/4 couchbas 0.00 B/s 114.16 K/s 0.00 % 0.27 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2613 be/4 couchbas 0.00 B/s 106.29 K/s 0.00 % 0.26 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2628 be/4 couchbas 0.00 B/s 78.73 K/s 0.00 % 0.24 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2625 be/4 couchbas 0.00 B/s 114.16 K/s 0.00 % 0.23 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2621 be/4 couchbas 0.00 B/s 102.35 K/s 0.00 % 0.23 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2617 be/4 couchbas 0.00 B/s 102.35 K/s 0.00 % 0.23 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2623 be/4 couchbas 0.00 B/s 94.48 K/s 0.00 % 0.22 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2622 be/4 couchbas 0.00 B/s 82.67 K/s 0.00 % 0.21 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2614 be/4 couchbas 0.00 B/s 74.80 K/s 0.00 % 0.20 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2627 be/4 couchbas 0.00 B/s 90.54 K/s 0.00 % 0.20 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2624 be/4 couchbas 0.00 B/s 70.86 K/s 0.00 % 0.16 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2616 be/4 couchbas 0.00 B/s 70.86 K/s 0.00 % 0.16 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2620 be/4 couchbas 0.00 B/s 62.99 K/s 0.00 % 0.15 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2626 be/4 couchbas 0.00 B/s 62.99 K/s 0.00 % 0.15 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
2618 be/4 couchbas 0.00 B/s 66.92 K/s 0.00 % 0.14 % beam.smp -A 16 -sbt u -P 327680 -K true -- -ro~base/var/lib/couchbase/couchbase-server.cookie"
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
Any ideas??
I disabled compaction and still having the high usage. Do replica sets cause a lot of disk usage?
No more so than the amount of data required to store the replica. What is your cluster's configuration? How many buckets? What's the replication number set to with each bucket?
I have 2 server nodes
I have 3 buckets (1 is couchbase, 2 are memcached)
I have 1 replica copy on the couchbase bucket
Couchbase Buckets
Create New Data Bucket
Bucket Name Nodes Item Count Ops/sec Disk Fetches/sec RAM Usage/Quota Disk Usage
pageviews 2 26316 452 0 12.8MB / 800MB 9.35GB
Memcached Buckets
Bucket Name Nodes Item Count Ops/sec Hit Ratio RAM Usage/Quota Disk Usage
default 2 4497 51 84.6% 362KB / 128MB 0B
memcachequick 2 4466180 2002 87.1% 300MB / 400MB 0B
My guess would be compaction. I've asked someone else to look at this thread and let us know how we can determine if compaction is underway.