replication doesn't replicate all items when failing over nodes from 6->5->4

Farshid Ghods added a comment - 12/Jul/11 5:37 PM seeing lot of suspend messages after the ndoe was failed over =============================================================================== ns_log: logging ns_orchestrator:6:Failed over ' ns_1@10.218.3.187 ': ok INFO REPORT <5881.229.0> 2011-07-06 04:27:10 =============================================================================== ns_log: suppressing duplicate log ns_orchestrator:6("Failed over ' ns_1@10.218.3.187 ': ok") because it's been seen 1 times in the past 271.853521 secs (last seen 271.853521 secs ago INFO REPORT <5881.221.0> 2011-07-06 04:27:10 =============================================================================== config change: {node,' ns_1@10.218.3.187 ',membership} -> inactiveFailed INFO REPORT <5881.221.0> 2011-07-06 04:27:10 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.257.0> 2011-07-06 04:27:10 =============================================================================== Pushing config INFO REPORT <5881.257.0> 2011-07-06 04:27:10 =============================================================================== Pushing config done INFO REPORT <5881.295.0> 2011-07-06 04:27:14 =============================================================================== memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.6.130 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.21.98 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.76.58.246 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.218.37.191 for 5.00 secs INFO REPORT <5881.295.0> 2011-07-06 04:27:19 =============================================================================== memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.21.98 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.218.37.191 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.76.58.246 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.6.130 for 5.00 secs INFO REPORT <5881.295.0> 2011-07-06 04:27:25 =============================================================================== memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.6.130 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.218.37.191 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.76.58.246 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.21.98 for 5.00 secs INFO REPORT <5881.295.0> 2011-07-06 04:27:31 =============================================================================== memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.76.58.246 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.6.130 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.218.37.191 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.21.98 for 5.00 secs INFO REPORT <5881.295.0> 2011-07-06 04:27:36 =============================================================================== memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.6.130 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.218.37.191 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.21.98 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.76.58.246 for 5.00 secs INFO REPORT <5881.295.0> 2011-07-06 04:27:41 =============================================================================== memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.21.98 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.218.37.191 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.82.6.130 for 5.00 secs memcached<0.295.0>: Suspend eq_tapq: replication_ns_1@10.76.58.246 for 5.00 secs

Farshid Ghods added a comment - 12/Jul/11 5:51 PM ran hash detail on all vbuckets on these nodes and found out that there are many vbuckets where replica item is 0 10.170.142.235 vbucket: 513 10.170.142.235 965 10.171.29.93 0 delta : 965 vbucket: 514 10.170.142.235 990 10.171.29.93 0 delta : 990 vbucket: 515 10.170.142.235 995 10.171.29.93 0 delta : 995 vbucket: 516 10.170.142.235 984 10.171.29.93 0 delta : 984 vbucket: 517 10.170.142.235 985 10.171.29.93 0 delta : 985 vbucket: 518 10.170.142.235 968 10.171.29.93 0 delta : 968 vbucket: 519 10.170.142.235 957 10.171.29.93 0 delta : 957 vbucket: 520 10.170.142.235 972 10.171.29.93 0 delta : 972 vbucket: 521 10.170.142.235 978 10.171.29.93 0 delta : 978 vbucket: 522 10.170.142.235 991 10.171.29.93 0 delta : 991 vbucket: 523 10.170.142.235 987 10.171.29.93 0 delta : 987 vbucket: 524 10.170.142.235 982 10.171.29.93 0 delta : 982 vbucket: 525 10.170.142.235 984 10.171.29.93 0 delta : 984 vbucket: 526 10.170.142.235 959 10.171.29.93 0 delta : 959 vbucket: 527 10.170.142.235 962 10.171.29.93 0 delta : 962 vbucket: 528 10.170.142.235 977 10.171.29.93 0 delta : 977 vbucket: 529 10.170.142.235 984 10.171.29.93 0 delta : 984 vbucket: 530 10.170.142.235 964 10.171.53.212 0 delta : 964 vbucket: 531 10.170.142.235 963 10.171.53.212 0 delta : 963 vbucket: 532 10.170.142.235 964 10.171.53.212 0 delta : 964 vbucket: 533 10.170.142.235 966 10.171.53.212 0 delta : 966 vbucket: 534 10.170.142.235 1000 10.171.53.212 0 delta : 1000 vbucket: 535 10.170.142.235 999 10.171.53.212 0 delta : 999 vbucket: 536 10.170.142.235 992 10.171.53.212 0 delta : 992 vbucket: 537 10.170.142.235 978 10.171.53.212 0 delta : 978 vbucket: 538 10.170.142.235 959 10.171.53.212 0 delta : 959 vbucket: 539 10.170.142.235 962 10.171.53.212 0 delta : 962 vbucket: 540 10.170.142.235 972 10.171.53.212 0 delta : 972 vbucket: 541 10.170.142.235 969 10.171.53.212 0 delta : 969 vbucket: 542 10.170.142.235 983 10.171.53.212 0 delta : 983 vbucket: 543 10.170.142.235 990 10.171.53.212 0 delta : 990 vbucket: 544 10.170.142.235 986 10.171.53.212 0 delta : 986 vbucket: 545 10.170.142.235 987 10.171.53.212 0 delta : 987 vbucket: 546 10.170.142.235 962 10.171.53.212 0 delta : 962 vbucket: 616 10.170.142.235 0 10.171.29.93 986 delta : 986 vbucket: 617 10.170.142.235 0 10.171.29.93 983 delta : 983 vbucket: 618 10.170.142.235 0 10.171.29.93 960 delta : 960 vbucket: 619 10.170.142.235 0 10.171.29.93 956 delta : 956 vbucket: 620 10.170.142.235 0 10.171.29.93 979 delta : 979 vbucket: 621 10.170.142.235 0 10.171.29.93 972 delta : 972 vbucket: 622 10.170.142.235 0 10.171.29.93 989 delta : 989 vbucket: 623 10.170.142.235 0 10.171.29.93 992 delta : 992 vbucket: 624 10.170.142.235 0 10.171.29.93 968 delta : 968 vbucket: 625 10.170.142.235 0 10.171.29.93 966 delta : 966 vbucket: 626 10.170.142.235 0 10.171.29.93 998 delta : 998 vbucket: 627 10.170.142.235 0 10.171.29.93 998 delta : 998 vbucket: 628 10.170.142.235 0 10.171.29.93 983 delta : 983 vbucket: 629 10.170.142.235 0 10.171.29.93 975 delta : 975 vbucket: 630 10.170.142.235 0 10.171.29.93 963 delta : 963 vbucket: 631 10.170.142.235 0 10.171.29.93 963 delta : 963 vbucket: 650 10.170.142.235 0 10.171.53.212 972 delta : 972 vbucket: 651 10.170.142.235 0 10.171.53.212 970 delta : 970 vbucket: 652 10.170.142.235 0 10.171.53.212 958 delta : 958 vbucket: 653 10.170.142.235 0 10.171.53.212 964 delta : 964 vbucket: 654 10.170.142.235 0 10.171.53.212 990 delta : 990 vbucket: 655 10.170.142.235 0 10.171.53.212 978 delta : 978 vbucket: 656 10.170.142.235 0 10.171.53.212 968 delta : 968 vbucket: 657 10.170.142.235 0 10.171.53.212 956 delta : 956 vbucket: 658 10.170.142.235 0 10.171.53.212 980 delta : 980 vbucket: 659 10.170.142.235 0 10.171.53.212 984 delta : 984 vbucket: 660 10.170.142.235 0 10.171.53.212 994 delta : 994 vbucket: 661 10.170.142.235 0 10.171.53.212 999 delta : 999 vbucket: 662 10.170.142.235 0 10.171.53.212 971 delta : 971 vbucket: 663 10.170.142.235 0 10.171.53.212 959 delta : 959 vbucket: 664 10.170.142.235 0 10.171.53.212 953 delta : 953 vbucket: 665 10.170.142.235 0 10.171.53.212 961 delta : 961 vbucket: 666 10.170.142.235 0 10.171.53.212 992 delta : 992 10.171.29.93 vbucket: 547 10.171.29.93 0 10.170.162.254 976 delta : 976 vbucket: 548 10.171.29.93 0 10.170.162.254 958 delta : 958 vbucket: 549 10.171.29.93 0 10.170.162.254 965 delta : 965 vbucket: 550 10.171.29.93 0 10.170.162.254 990 delta : 990 vbucket: 551 10.171.29.93 0 10.170.162.254 986 delta : 986 vbucket: 552 10.171.29.93 0 10.170.162.254 997 delta : 997 vbucket: 553 10.171.29.93 0 10.170.162.254 1001 delta : 1001 vbucket: 554 10.171.29.93 0 10.170.162.254 973 delta : 973 vbucket: 555 10.171.29.93 0 10.170.162.254 960 delta : 960 vbucket: 556 10.171.29.93 0 10.170.162.254 966 delta : 966 vbucket: 557 10.171.29.93 0 10.170.162.254 954 delta : 954 vbucket: 558 10.171.29.93 0 10.170.162.254 981 delta : 981 vbucket: 559 10.171.29.93 0 10.170.162.254 985 delta : 985 vbucket: 560 10.171.29.93 0 10.170.162.254 958 delta : 958 vbucket: 561 10.171.29.93 0 10.170.162.254 963 delta : 963 vbucket: 562 10.171.29.93 0 10.170.162.254 989 delta : 989 vbucket: 563 10.171.29.93 0 10.170.162.254 975 delta : 975 vbucket: 632 10.171.29.93 968 10.170.162.254 0 delta : 968 vbucket: 633 10.171.29.93 970 10.170.162.254 0 delta : 970 vbucket: 634 10.171.29.93 992 10.170.162.254 0 delta : 992 vbucket: 635 10.171.29.93 984 10.170.162.254 0 delta : 984 vbucket: 636 10.171.29.93 981 10.170.162.254 0 delta : 981 vbucket: 637 10.171.29.93 993 10.170.162.254 0 delta : 993 vbucket: 638 10.171.29.93 964 10.170.162.254 0 delta : 964 vbucket: 639 10.171.29.93 961 10.170.162.254 0 delta : 961 vbucket: 640 10.171.29.93 990 10.170.162.254 0 delta : 990 vbucket: 641 10.171.29.93 996 10.170.162.254 0 delta : 996 vbucket: 642 10.171.29.93 959 10.170.162.254 0 delta : 959 vbucket: 643 10.171.29.93 968 10.170.162.254 0 delta : 968 vbucket: 644 10.171.29.93 967 10.170.162.254 0 delta : 967 vbucket: 645 10.171.29.93 962 10.170.162.254 0 delta : 962 vbucket: 646 10.171.29.93 978 10.170.162.254 0 delta : 978 vbucket: 647 10.171.29.93 986 10.170.162.254 0 delta : 986 vbucket: 648 10.171.29.93 996 10.170.162.254 0 delta : 996 vbucket: 649 10.171.29.93 992 10.170.162.254 0 delta : 992 10.171.53.212 vbucket: 564 10.171.53.212 0 10.170.162.254 993 delta : 993 vbucket: 565 10.171.53.212 0 10.170.162.254 989 delta : 989 vbucket: 566 10.171.53.212 0 10.170.162.254 974 delta : 974 vbucket: 567 10.171.53.212 0 10.170.162.254 972 delta : 972 vbucket: 568 10.171.53.212 0 10.170.162.254 968 delta : 968 vbucket: 569 10.171.53.212 0 10.170.162.254 963 delta : 963 vbucket: 570 10.171.53.212 0 10.170.162.254 981 delta : 981 vbucket: 571 10.171.53.212 0 10.170.162.254 988 delta : 988 vbucket: 572 10.171.53.212 0 10.170.162.254 991 delta : 991 vbucket: 573 10.171.53.212 0 10.170.162.254 997 delta : 997 vbucket: 574 10.171.53.212 0 10.170.162.254 958 delta : 958 vbucket: 575 10.171.53.212 0 10.170.162.254 967 delta : 967 vbucket: 576 10.171.53.212 0 10.170.162.254 961 delta : 961 vbucket: 577 10.171.53.212 0 10.170.162.254 953 delta : 953 vbucket: 578 10.171.53.212 0 10.170.162.254 989 delta : 989 vbucket: 579 10.171.53.212 0 10.170.162.254 992 delta : 992 vbucket: 580 10.171.53.212 0 10.170.162.254 989 delta : 989 vbucket: 667 10.171.53.212 989 10.170.162.254 0 delta : 989 vbucket: 668 10.171.53.212 987 10.170.162.254 0 delta : 987 vbucket: 669 10.171.53.212 989 10.170.162.254 0 delta : 989 vbucket: 670 10.171.53.212 961 10.170.162.254 0 delta : 961 vbucket: 671 10.171.53.212 977 10.170.162.254 0 delta : 977 vbucket: 672 10.171.53.212 970 10.170.162.254 0 delta : 970 vbucket: 673 10.171.53.212 968 10.170.162.254 0 delta : 968 vbucket: 674 10.171.53.212 984 10.170.162.254 0 delta : 984 vbucket: 675 10.171.53.212 992 10.170.162.254 0 delta : 992 vbucket: 676 10.171.53.212 993 10.170.162.254 0 delta : 993 vbucket: 677 10.171.53.212 980 10.170.162.254 0 delta : 980 vbucket: 678 10.171.53.212 961 10.170.162.254 0 delta : 961 vbucket: 679 10.171.53.212 964 10.170.162.254 0 delta : 964 vbucket: 680 10.171.53.212 966 10.170.162.254 0 delta : 966 vbucket: 681 10.171.53.212 969 10.170.162.254 0 delta : 969 vbucket: 682 10.171.53.212 999 10.170.162.254 0 delta : 999 vbucket: 683 10.171.53.212 998 10.170.162.254 0 delta : 998

Farshid Ghods added a comment - 12/Jul/11 5:53 PM for instance bucket 513 master node has 965 items but replica node has 0 items 10.170.142.235 vbucket: 513 10.170.142.235 965 10.171.29.93 0 delta : 965

Farshid Ghods added a comment - 12/Jul/11 6:28 PM [ root@ip-10-171-29-93 ~]# /opt/membase/bin/mbstats localhost:11211 all | grep curr_  curr_connections: 55  curr_items: 965828  curr_items_tot: 1798804  vb_active_curr_items: 965828  vb_pending_curr_items: 0  vb_replica_curr_items: 832976

Chiyoung Seo added a comment - 14/Jul/11 6:06 PM I think this is not a bug because failover without rebalance does not establish the replication channel.