Membase instance becomes unstable and unreacheable after an hour

Hi,

Just had some weird problems with our production cache cluster which we can't seem to find any explanations for. Almost exactly an hour after deploying some changes to the cluster (going to a two-node cluster from a single node deployment, and adding a membase bucket that has 1X replication turned on) one of the servers (the new node) started behaving strangely:
- its CPU utilization shot up to 50% from around 5-8% and stayed there, logging onto the instance we were able to observe that all that CPU utilization was attributed to the membase.exe process
- as far as we could tell, this node was not functioning, our code couldn't talk to it, but it shows up as normal in the membase console however
- there were quite a few error messages in the Log at the exactly same time (10.212.203.146 is the bad server), such as this message:
Control connection to memcached on 'ns_1@10.212.203.146' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
stats_recv,
4},
{mc_client_binary,
stats,
4},
{ns_memcached,
handle_call,
3},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} (repeated 5 times)
- when we run browse_log.bat we could a lot of errors :
ns_1@10.212.203.146:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,
system_limit}}}, retrying.

Some more context around the deployment:
- everything's deployed in Amazon EC2
- they're running on high CPU medium instances
- the get/store operations are all performed with cas
- average size of the objects stored in the membase bucket is around 44k and we had around 1.8k of these objects in the bucket at the time
- there were at most 300-400 ops/s across all the buckets (4 memcached, 1 membase) on the cluster
- disk space was not full, there was more than 17GB available

Since then, we have replaced the instances and reconstructed the cluster and everything seems to work so far (it's been an hour and half, so fingers crossed!), but we would really like to find out what went wrong the first time around and if we're not interacting with the cache correctly. Mind you, this is the first time we've come across this behaviour, having worked with multi-node deployments of NorthScale, and various versions of Membase with different topologies and mixture of membase and memcached buckets...

Any help would be much appreciated!

Thanks,