2nd server issue
Hi. I previously had version 1.6.5 installed on 2 servers (my local machine and a seperate machine on the network). All worked well for a while but then the instance on my local machine kept crashing / CPU spiking so I upgraded to version 1.7 (both servers). This went alright, however I don't seem to be able to remove my local machine from the cluster (FYI my local machine just joined the cluster, it was setup on the other machine). Any attempt to rebalance and/or remove then rebalance results in the error:
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@MY.LOCAL.IP.ADDRESS']}
(repeated 1 times)
As such I can never seem to remove my local machine from the cluster and some cache items, according to the web interface appear to be stored, but trying to retrieve them returns nothing. I assume because it's trying to use my local for storage, but of course it can't retrieve from there.
Any ideas why the rebalance is failing and how I can remove a server that won't rebalance?
Cheers
Hey farshid. Thanks for that, that does seem to have removed it.
1- Different OS (both on windows, just different versions), same network
2- nope, no IP change
The other thing I've just come across, membase doesn't seem to be having success storing some of my objects (C#). Are there any restrictions around the data / keys it can store?
johnnyboy,
>>The other thing I've just come across, membase doesn't seem to be having success storing some of my objects (C#). Are there any restrictions around the data / keys it >>can store?
you must be using Enyim client for set/getting keys.
do you know what error you get when trying to set and item.
since one of the nodes seems to be inaccessible you are probably seeing "not my vbucket" or other errors.
>>1- Different OS (both on windows, just different versions), same network
in that case i recommed setting up the cluster on two homogenous nodes. also failing over the node and rebalancing it out should get you out of the state where you are not able to remove the node or set some of the keys.
Correct, Enyim - should I not be using this? There isn't any error it's just when using Set it returns false (as in it couldn't save) and attempting to Get using the same key returns null. The basic web interface log doesn't have any errors about this, is there another log somewhere with more info? If I perform two cache saves concurrently, one a basic string and the other my custom object the string will cache but my custom object will not.
Sorry farshid, you'll have to clarify that for me - a cluster on two homogenous nodes?
I've failed over the node and it's no longer appearing in the server nodes list - I assume it's gone.
>>Sorry farshid, you'll have to clarify that for me - a cluster on two homogenous nodes?
same OS , same architecture , ( e.g both windows 64-it 2008 R2 )
>>I've failed over the node and it's no longer appearing in the server nodes list - I assume it's gone.
yes. you have successfully removed this node from the cluster.
>>Correct, Enyim - should I not be using this? There isn't any error it's just when using Set it returns false (as in it couldn't save) and attempting to Get using the same key >>returns null. The basic web interface log doesn't have any errors about this, is there another log somewhere with more info? If I perform two cache saves concurrently, one >>a basic string and the other my custom object the string will cache but my custom object will not.
seems like a client error ( probably wrong vbucket or wrong key ...)
can you also post the part of the code where you instantiate and set an item using Enyim client ?
I'm away from my PC just now so forgive any syntax errors or the sort but essentially my test code is:
MembaseClient membaseClient = new MembaseClient();
string key1 = "Key1";
string key2 = "Key2";
object o1 = new MyEntity();
object o2 = "12345";
membaseClient.Store(Set, key1, o1, new TimeSpan(0, 1, 0)); // returns false
membaseClient.Store(Set, key2, o2, new TimeSpan(0, 1, 0)); // returns true
Can you check if the class MyEntity is marked as serializable?
Yeah that was the problem, didn't realise that every object had to be explicitly marked as serialable.
Thanks guys
>>Any ideas why the rebalance is failing and how I can remove a server that won't rebalance?
you can try failing over the node and then rebalance out that node.
1- are these two nodes on the same OS and same network.
2- did your local machine's ip address have changed since you have added the node to the cluster?