How do I limit the log size over /opt/membase/var/lib/membase/mnesia/LATEST.log

I believe this is unexpected behavior, but I'll double check with some colleagues.

It is unexpected behavior. And in order to help you we need more information about your system. What OS? How many buckets?

Best would be if you file a bug in jira.couchbase.com with as much information as possible. Most useful would be so called diag. It can be optained by clicking link at left-top of logs in Logs section.

The operating system is Red Hat Enterprise Linux Server release 5.6 (Tikanga).
The membase package is membase-server-community_x86_64_1.7.1.1.rpm.
There is only one bucket "default" in membase cluster at that time.
The LATEST.log was more than 7G before deletion.

And it seems like jira.couchbase.com is not valid.

Thank you.
yw

Here's the right URL: http://www.couchbase.com/issues/secure/Dashboard.jspa

a jira bug has been filed (MB-4795). Thank you.

yw

Thank you for filing bug. Can you please attach result of /opt/couchbase/bin/mbcollect_info ? That would provide us more info about your particular case. Logs and generic system info.

Also we think it might be related to huge IO load of system. And would like you to confirm or decline that.

So was system highly loaded with inserts or updates during that period ?

The mbcollect_info has ran and uploaded to jira. And actually no hugeIO has run at the time.
Question for this LATEST.log file: is it generated by SQLlite? Can we use the following command to clean it?

mbdbmaint [--vacuum] [--backupto=] [--port=11210]

Is there anyway to set up a size to auto rolling the log?

Thank you
-yw

No those file are used/created-by mnesia. That's erlang's database we're currently using to store out stats.

And my co-worker have found the following bug posted against erlang, which exactly describe the issue we had. Here is the link:

http://erlang.org/pipermail/erlang-bugs/2011-July/002511.html

Here is the erlang version that comes with the membase distribution:
{"OTP APN 181 01","R14B02"}

Here is the release notes from erlang amnesia release.

for OTP-9551

Dump the log even if no transactions have been invoked on local node, otherwise the log will grow forever with decisions from the other nodes who have tables on disk.

And the fix is bundled in R14B04.

The latest couchbase1.8 contains erlang R14B03 that doesn't contain the fix.

Thanks for finding it. I'll double check, but it's possible that this fix does not address our case. We're not using mnesia in distributed fashion.

Based on the description of the bug, it seems like one of the membase server in the cluster never had any hit (transaction), which causes the LATEST.log grow forever.
it seems like that the method used to calculate which node the key reside is not balanced across the nodes in membase.

This issue has nothing to do with key distribution. We use mnesia _only_ to archive node's stats. And each node does it's own stats archiving independently from other nodes.

Regarding this particular erlang bug fix I'll double check.

Was this ever fixed?

I just had to stop the membase server (1.7.2r-20-g6604356) so I can delete LATEST.log. It got to 10GB and the server ran out of disk space. I could not find a way to do this without stop the server

The dataset is about 200MB in total with of approx 271.1K items and an average of 100 operations per second so it doesn't really make sense. The server does not seem to be "overloaded"

We've never explicitly fixed this problem because we still don't understand how this happens. There's good chance however that this is erlang bug and that this bug is fixed in later version of erlang, which later releases of couchbase ship.

In a production environment, would it be possible to upgrade erlang (or any other component for that matter) without updating the entire membase server?

It is possible in principle (say, if you're building from source) but not sure if it's practical to try that with our binary packages. Nobody ever tried that AFAIK