Rebalance exited with reason {detected_nodes_change,

Tommie McAfee added a comment - 03/May/12 5:39 PM test case: ./testrunner -i resources/jenkins/centos-64-7node-rebalance.ini -t rebalancetests.IncrementalRebalanceOut.test_load,replica=1,delete-ratio=0,expiry-ratio=0

Dipti Borkar added a comment - 03/May/12 7:58 PM Hi Aliaksey A, based on Farshid's feedback this is currently blocking DP5 testing. Can you please take a look?

Aliaksey Artamonau added a comment - 04/May/12 6:18 AM Interestingly, we finally reproduced this "corrupted external term" issue: [error_logger:error] [2012-05-03 13:25:48] [ ns_1@10.3.121.152 :error_logger:ale_error_logger_handler:log_msg:76] ' ns_1@10.3.121.152 ' got a corrupted external term from ' ns_1@10.3.121.148 ' on distribution channel 17660 <<...,104,2,104,2,114,0,3,82,0,2,0,3,35,25,0,0,0,70,0,0,0,0,82,3,108,0,0,0,1,104,2,107,0,8,98,117,99,107,101,116,45,48,108,0,0,0,10,104,2,107,0,13,50,57,49,52,49,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,57,49,48,53,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,57,48,55,51,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,57,48,51,55,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,56,57,55,51,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,56,57,54,50,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,56,57,51,55,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,56,57,50,54,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,57,49,53,57,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,104,2,107,0,13,50,57,49,52,56,45,53,97,98,53,100,102,54,108,0,0,0,1,104,2,82,2,70,127,240,0,0,0,0,0,0,106,106,106>> ATOM_CACHE_REF translations: 0=' ns_1@10.3.121.152 ', 1='', 2=ops, 3=' ns_1@10.3.121.148 '

Aliaksey Artamonau added a comment - 04/May/12 6:12 PM I was able to find out what exactly destination node didn't like in this packet. It was float value representing positive infinity. Which is very weird because it's claimed that there's no way to introduce such a value from erlang. So I need to investigate it further.

Aleksey Kondratenko added a comment - 04/May/12 10:02 PM Farshid, we'll need to get config.log from inside tree where our erlang vm is built. Send it to us separately.

Aleksey Kondratenko added a comment - 07/May/12 1:14 PM Also please add in what environment you're observing those. Is that specific to Xen or any other speicific type of virtualization or it's reproducible on everything ?

Aleksey Kondratenko added a comment - 07/May/12 3:20 PM Attaching erl_arith intermediate assembly with all options and exactly same compiler as built by our 2.0 rhel 5 builder. If there are some issues with infinity being missed, erl_arith is most likely place.

Aleksey Kondratenko added a comment - 07/May/12 4:57 PM Farshid, this looks like either environment or compiler or erlang vm issue. Just in case lets try building erlang with the following ./configure options: --disable-hipe --disable-fp-exceptions I've verified that indeed this gives different code which is hopefully more robust.

Peter Wansch added a comment - 19/Jun/12 7:48 PM Farshid, can you verify if this still happens or if it's an environment/setup issue?