Delete all items from storage bucket

The flush_all followed by a safe restart should have actually removed them from the system. In fact, we do test this very situation, though not directly through the telnet interface. We'll look into this to see if we can see where the issue is.

As a workaround, moving the "config" and "data" under /opt/NorthScale/1.6.0beta1 out of the way should remove the data and revert the node to an unconfigured state.

Thanks for reporting it!

> As a workaround, moving the "config" and "data" under
> /opt/NorthScale/1.6.0beta1 out of the way should remove
> the data and revert the node to an unconfigured state.

thanks.
can you give me some more info about the workaround?

under /opt/NorthScale/1.6.0beta, I see a "data" directory, but no "config" (file or directory). Where is the config I need to move?
also, when you say "move out of the way", do you mean just deleting/moving the entire directory? If so, does this get recreated at startup, or do I need to do anything else to get an empty bucket?

-Corey

Corey, I tried out the flush_all command, restarted the server and my data is gone.

I definitely want to dig into this with you to make sure we don't have a bug somewhere (which is entirely possible).

A couple questions:
-How many nodes in the cluster do you have?
-Are you using a "safe" restart (/etc/init.d/northscale restart)?

Perry

> How many nodes in the cluster do you have?

single node

> Are you using a "safe" restart (/etc/init.d/northscale restart)?

yes

... as soon as I restart the server, I see server memory go down, but then it starts filling up again. while the data is filling, the telnet interface is not available (all commands return "SERVER_ERROR proxy write to downstream"). Once all of the data is back (in my case, 7GB of it), I can use the telnet interface and things work normally (although my mem is nearly full).

-Corey

more info:
I just ran a small test with < 1000 items in the bucket, and the flush_all -> restart seemed to clear the data. However, if I seed a large data set in the bucket (few hundred k items), then I run into the problem.

Perhaps I am not letting the flush_all command finish and it needs time with such a large data set? (I just wait until the telnet interface comes back with "OK" before I do a server restart.) Could that be a possibility? I will run more tests to confirm the behavior.

-Corey

That could definitely be a possibility, but the graceful shutdown is supposed to force all the data to be cleaned up.

I'll try to reproduce on a much larger scale as well.

Your testing is greatly appreciated!

Perry

related observations:
when I run flush_all on a large data set (500k items consuming 7GB), one of my CPU's spikes to 100% and stays pegged (I killed it after 20 mins).

the process consuming all of the CPU is:
./bin/memcached/memcached -p 11210 -X ./bin/memcached/stdin_term_handler.so -E ./bin/ep_engine/ep.so -B binary -r -e vb0=false;ht_size=12289;ht_locks=23;dbname=/opt/NorthScale/1.6.0beta1/data/ns_1/default

this process also remains after I shutdown the server.

-Corey

Thanks Corey, I think we're on a path to reproduce this here as well. I will keep you updated with any progress.

Thanks again.

Perry