[Error Logs] 'verify_report_long_call' entries in couchbase logs

Couchbase Server
1

Hello,

Our couchbase cluster is running fine, but we always have a lot of ‘verify_report_long_call’ entries in couchbase error logs.

  • What does this really mean and imply in terms of performance/stability/data integrity ?
  • What did we do wrong to have all those log entries and what should we change/tune to get rid of it ?
  • To the community : Do you have those log entries and at what pace ?
  • To get a rough idea of the pace this happens :
cd /opt/couchbase/var/lib/couchbase/logs  # Or cd wherever your couchbase logs are

awk -F"[,: ]" '$0~/ns_memcached:verify_report_long_call/{hour[$3]++} END{for (i in hour) {print i" "hour[i]}}' ./error.*

Log entries details :

  • We have an average 300 log entries of this type every hour, and this rate is pretty much stable for the past few months
  • It sometimes spikes above 300/Hour and can go up to 1 000, but this is really rare
  • Detailed logs shows it concerns the following 5 ‘actions’ (% in col 1) :
    • 3% list_vbuckets
    • 44% {stats,<<>>}
    • 3% {stats,<<"tapagg
    • 21% {stats,<<“timings”>>}
    • 29% topkeys

Our conf :

  • Couchbase 2.1.1 CE
  • 6 nodes (Running inside distinct openVZ containers on 3 Physical Servers)
  • 2 replicas per Bucket
  • 5 Buckets
  • 50% GETs 50% SETs on most buckets
  • 100% INCRs on 1 bucket
  • A couple of views (Called less than once per minute)