Performance loss with PHP setMulti Operation over existing keys
I'm running several hundred million documents into couchbase using php and the setMulti feature 10,000 at a time. If I run the operation on an empty bucket, I get around 600,000 ops. If I re-run that operation on a bucket containing the key/values from the first run my ops only runs at around 150,000. I've got a 6 Node cluster @ 8 core 2.0ghz and 64GB Ram. Disks formatted to ext4. Any ideas what would cause this?
Me too.
kneeoh: I have a weird issue with a very large bucket. It doesn't seem to matter what size I make the bucket the ram usage always seems to creep up to near the quota.
[5:42pm] kneeoh: when I first load the data on a 146GB bucket the space used is around 73GB. I walk away for a few hours (NOTHING) is going on with activity on the system as it's 100% dev and only I use it. And now there's 132GB / 146GB used.
[5:43pm] kneeoh: the same was true for this bucket when I had it at 96GB
[5:43pm] kneeoh: it would run up to 90 or so
[5:44pm] kneeoh: The bigger issue is that I can not run any set operations over this bucket when it gets to this state. All I get are temporary failures from the client library
It's worse than I thought..It won't even let me setMulti over the existing keys. Debug reveals: Temporary failure. Try again later every time a set operation is performed. Does this mean I need memory to be 2 times the size of the data set? 1x to hold it and 2x to store the keys being over written?