Membase default data bucket not returning data.
I have membase version 1.6.5. We are using single node membase for generating tiles of the order of 10M entries. Its accessed simultaneously by multiple machine for storing the intermediate/final results. Membase worked fine till then. Now when 11 million entries have been loaded, we needed to copy the data to another computer. So we start pulling the data out of membase. This worked fine uptil 8 M entries and after that the membase started failing. It failed to return the data. When I check at the logs its says:
Control connection to memcached on 'ns_1@10.3.0.15' disconnected: {{badmatch,
{error,
closed}},
[{mc_client_binary,
cmd_binary_vocal_recv,
5},
{mc_client_binary,
select_bucket,
2},
{ns_memcached,
ensure_bucket,
2},
{ns_memcached,
handle_info,
2},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} (repeated 1 times)
So this morning I decided to restart the membase. I can see one out of the 3 data buckets was loaded. 2 Data buckets never loaded. These two data buckets are very critical for us. Although the STAT ep_warmup: true.
I have gathered the full error report. Where can i attach it? Any help will be highly appreciated.
Thanks,
-Sandeep
First off, I would highly recommend upgrading this to 1.7.
Secondly, you can monitor the progress of warmup by watching the "ep_warmed_up" stat which will tell you how many items it has done, and the "ep_warmup_time" will tell you how long it has been running for (or how long it ran once completed). The "ep_warmup" stat is just telling you that the software knows it needs to warmup.
Perry
Now the bucket with 1654044 entries was loaded in 5552 seconds.
But the bucket with 10296881 entries hasnt come up yet