Repeated failures creating bucket

Perry Krug added a comment - 04/Dec/12 11:26 AM Same action, different error being reported:  Save request failed because of timeout. And the UI logs show: Server error during processing: ["web request failed", {path,"/pools/default/buckets"}, {type,exit}, {what, {{badarg, [{io_lib,format, ["Best-effort check for presense of bucket failed to be made on following nodes: ~p", [' ns_1@10.145.129.217 ', ' ns_1@10.145.169.48 ', ' ns_1@10.145.171.199 ']]}, {'ale_logger-ns_server',warn,5}, {ns_orchestrator,idle,3}, {gen_fsm,handle_msg,7}, {proc_lib,init_p_do_apply,3}]}, {gen_fsm,sync_send_event, [{global,ns_orchestrator}, {create_bucket,membase,"AbRegNums", [{num_replicas,1}, {replica_index,false}, {ram_quota,38797312000}, {auth_type,sasl}, {sasl_password,[]}, {autocompaction,false}, {flush_enabled,false}]}, infinity]}}}, {trace, [{gen_fsm,sync_send_event,3}, {menelaus_web_buckets,do_bucket_create,2}, {menelaus_web_buckets, handle_bucket_create,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}]

Farshid Ghods added a comment - 04/Dec/12 11:32 AM bringing this back to 2.0 since this needs triaging today!

Farshid Ghods added a comment - 04/Dec/12 1:52 PM Not sure what state the cluster is in that is causing this, but I have been unable to create new buckets on a 20 node cluster. Occasionally it will work, but most of the the time UI errors out with: Unexpected server error, request logged. and the UI log shows: Server error during processing: ["web request failed", {path,"/pools/default/buckets"}, {type,exit}, {what, {noproc, {gen_fsm,sync_send_event, [{global,ns_orchestrator}, {create_bucket,membase,"test", [{num_replicas,1}, {replica_index,false}, {ram_quota,65536000000}, {auth_type,sasl}, {sasl_password,[]}, {autocompaction,false}, {flush_enabled,false}]}, infinity]}}}, {trace, [{gen_fsm,sync_send_event,3}, {menelaus_web_buckets,do_bucket_create,2}, {menelaus_web_buckets, handle_bucket_create,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}] I imagine it has something to do with the fact that the cluster is currently compacting a re-replicating a very large bucket, but there's no indication of what the problem is nor any reason why a simple bucket creation should fail here. Logs for the whole cluster are at: https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-1 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-2 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-3 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-4 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-5 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-6 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-7 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-8 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-19 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-10 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-11 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-12 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-13 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-14 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-15 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-16 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-17 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-18 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-19 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-20

Farshid Ghods added a comment - 04/Dec/12 1:52 PM Not sure what state the cluster is in that is causing this, but I have been unable to create new buckets on a 20 node cluster. Occasionally it will work, but most of the the time UI errors out with: Unexpected server error, request logged. and the UI log shows: Server error during processing: ["web request failed", {path,"/pools/default/buckets"}, {type,exit}, {what, {noproc, {gen_fsm,sync_send_event, [{global,ns_orchestrator}, {create_bucket,membase,"test", [{num_replicas,1}, {replica_index,false}, {ram_quota,65536000000}, {auth_type,sasl}, {sasl_password,[]}, {autocompaction,false}, {flush_enabled,false}]}, infinity]}}}, {trace, [{gen_fsm,sync_send_event,3}, {menelaus_web_buckets,do_bucket_create,2}, {menelaus_web_buckets, handle_bucket_create,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}] I imagine it has something to do with the fact that the cluster is currently compacting a re-replicating a very large bucket, but there's no indication of what the problem is nor any reason why a simple bucket creation should fail here. Logs for the whole cluster are at: https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-1 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-2 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-3 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-4 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-5 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-6 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-7 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-8 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-19 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-10 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-11 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-12 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-13 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-14 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-15 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-16 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-17 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-18 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-19 https://s3.amazonaws.com/customers.couchbase.com/viber/bucket_creation/AMIR-DB-20

Aleksey Kondratenko added a comment - 04/Dec/12 3:50 PM Seeing this in dmesg: md/raid10:md127: not clean -- starting background reconstruction md/raid10:md127: active with 6 out of 6 devices md127: detected capacity change from 0 to 644240375808 md: resync of RAID array md127 md: minimum _guaranteed_ speed: 1000 KB/sec/disk. md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync. md: using 128k window, over a total of 629140992k. continuing looking...

Aleksey Kondratenko added a comment - 04/Dec/12 3:52 PM actually that was a bit too quick. I'm seeing resync done message almost immediately. So looks like they have 2 raid 10 devices on each on top of 6 EBS volumes

Aleksey Kondratenko added a comment - 04/Dec/12 3:55 PM And they're using XFS which is not seemingly much tested option FS-wise and w.r.t. our fsync-hunger

Perry Krug added a comment - 05/Dec/12 4:14 AM Thanks Alk, any idea why this just started popping up now? Do we officially not support XFS for the config directory?

Aleksey Kondratenko added a comment - 06/Dec/12 12:39 PM I was mentioning XFS in context of data volume. I don't think we officially refuse to support it, but we know that our DB right now is a bit excessively fsync-heavy. And those fsyncs currently update files metadata exercising filesystem metadata journal-ing efficiency. There's rumors that XFS is not as efficient on this code path as ext4. But I don't think this is related to this particular problem.

Aleksey Kondratenko added a comment - 06/Dec/12 12:41 PM Just seeing lots of timeouts here and there. Duplicate of multiple timeout bugs we already have. As far as I known we currently believe that stopping xdcr will heal this perhaps after some minutes of waiting.

Perry Krug added a comment - 06/Dec/12 6:08 PM Do we have bug(s) to track these timeout issues? There was no XDCR going on at the time.

Farshid Ghods added a comment - 10/Dec/12 11:30 AM deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )

Aleksey Kondratenko added a comment - 12/Dec/12 6:17 PM Indeed there's no XDCR. We'll have to assume it's caused by MB-6595 . Which is queued for investigation.

Perry Krug added a comment - 19/Dec/12 10:28 AM dupe of mb-7111