Hi Support Team,
We faced one of the couchbase node failedover in cluster , which we need to manually add and we performed rebalance.
We would like to know the root cause of the failover issue.
Please let us know how we can verify the root cause of the issue? Which logfile will have right details?
-
What versions of the software are you running - Membase/Couchbase Server, moxi, and client drivers? 2.0.1 community edition (build-170)
-
Operating system version, architecture (32-bit or 64-bit) and deployment (physical hardware, Amazon EC2, RightScale, etc.) Linux glcapp-wc-a1p.sys.comcast.net 2.6.32-279.22.1.el6.x86_64 #1 SMP Wed Feb 6 03:10:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
-
Number of nodes in the cluster, how much physical RAM in each node, and per-node RAM allocated to Couchbase Server : 4 node , 94gb ram on each node,
-
What steps led to the failure or error? No Idea
-
Information around whether this is something that has worked successfully in the past and if so what has changed in the environment since the last successful operation? There is no recent changes
-
Provide us with a current snapshot of logs taken from each node of the system and uploaded to our support system via the instructions below
· Node (‘ns_1@...**’) was automatically failovered.
· [{last_heard,{1467,741141,461246}},
· {outgoing_replications_safeness_level,[{“customerdb”,green}]},
· {incoming_replications_conf_hashes,[{“customerdb”,}]},
· {active_buckets,[“customerdb”]},
· {ready_buckets,},
· {local_tasks,},
· {memory,[{total,730980608},
· {processes,503678984},
· {processes_used,502451840},
· {system,227301624},
· {atom,1675105},
· {atom_used,1648934},
· {binary,25652160},
· {code,16431806},
· {ets,141513432}]},
· {system_memory_data,[{system_total_memory,101402648576},
· {free_swap,16722644992},
· {total_swap,17179860992},
· {cached_memory,14753308672},
· {buffered_memory,64200704},
· {free_memory,23321243648},
· {total_memory,101402648576}]}, {node_storage_conf,[{db_path,"/app/app/couchbase/data"},
· {index_path,"/app/app/couchbase/data"}]},
· {statistics,[{wall_clock,{2804501661,285}},
· {context_switches,{26502565031,0}},
· {garbage_collection,{1956687139,10097591388395,0}},
· {io,{{input,1994359133028},{output,1986144407549}}},
· {reductions,{3716814054952,511405}},
· {run_queue,0},
· {runtime,{966201260,100}}]}, {system_stats,[{cpu_utilization_rate,13.54292623941959},
· {swap_total,17179860992},
· {swap_used,457216000}]}, {interesting_stats,[{couch_docs_actual_disk_size,40058438371},
· {couch_docs_data_size,32343125247},
· {couch_views_actual_disk_size,142804149},
· {couch_views_data_size,104328277},
· {curr_items,0},
· {curr_items_tot,0},
· {mem_used,500571712},
· {vb_replica_curr_items,0}]}, {cluster_compatibility_version,131072},
· {version,[{public_key,“0.13”},
· {lhttpc,“1.3.0”},
· {ale,“8cffe61”},
· {os_mon,“2.2.7”},
· {couch_set_view,“1.2.0a-8352437-git”},
· {mnesia,“4.5”},
· {inets,“5.7.1”},
· {couch,“1.2.0a-8352437-git”},
· {mapreduce,“1.0.0”},
· {couch_index_merger,“1.2.0a-8352437-git”},
· {kernel,“2.14.5”},
· {crypto,“2.0.4”},
· {ssl,“4.1.6”},
· {sasl,“2.1.10”},
· {couch_view_parser,“1.0.0”},
· {ns_server,“2.0.1-170-rel-community”},
· {mochiweb,“1.4.1”},
· {oauth,“7d85d3ef”},
· {stdlib,“1.17.5”}]},
· {supported_compat_version,[2,0]},
· {system_arch,“x86_64-unknown-linux-gnu”},
· {wall_clock,2804501},
· {memory_data,{101402648576,77670584320,{<15377.6.0>,47741960}}},
· {disk_data,[{"/",10321208,29},
· {"/dev/shm",49513012,0},
· {"/app",495419208,17},
· {"/boot",516040,18},
· {"/opt",16513960,17},
· {"/var",15481840,9}]},
· {meminfo,<<“MemTotal: 99026024 kB\nMemFree: 22774768 kB\nBuffers: 62696 kB\nCached: 14407528 kB\nSwapCached: 65996 kB\nActive: 56435768 kB\nInactive: 16476548 kB\nActive(anon): 55176036 kB\nInactive(anon): 3308224 kB\nActive(file): 1259732 kB\nInactive(file): 13168324 kB\nUnevictable: 1337156 kB\nMlocked: 0 kB\nSwapTotal: 16777208 kB\nSwapFree: 16330708 kB\nDirty: 3672 kB\nWriteback: 0 kB\nAnonPages: 59719992 kB\nMapped: 62848 kB\nShmem: 240 kB\nSlab: 773944 kB\nSReclaimable: 672048 kB\nSUnreclaim: 101896 kB\nKernelStack: 34656 kB\nPageTables: 142296 kB\nNFS_Unstable: 0 kB\nBounce: 0 kB\nWritebackTmp: 0 kB\nCommitLimit: 66290220 kB\nCommitted_AS: 71289396 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed: 489900 kB\nVmallocChunk: 34308510088 kB\nHardwareCorrupted: 0 kB\nAnonHugePages: 57827328 kB\nHugePages_Total: 0\nHugePages_Free: 0\nHugePages_Rsvd: 0\nHugePages_Surp: 0\nHugepagesize: 2048 kB\nDirectMap4k: 8056 kB\nDirectMap2M: 3102720 kB\nDirectMap1G: 97517568 kB\n”>>}]