Couchbase reporting ?s

You should be able to since the Admin interface shows you that as well, let me find the syntax for you and add it to this reply.

Awesome thanks - I look forward to it.

Bonus Questions

1.) Can I change the length of time that the report is run over. Ex I was to see how many ops/sec key have over 5 mins, it seems like the Web Admin interface refreshes every few seconds.

2) How can I delete keys from couchbase once I identify that they haven't done many ops?

Did you see this: http://www.couchbase.com/docs/couchbase-manual-1.8/rest-api-bucket-stats..., if you include the since timestamp (haveTStamp) parameter you can get the hottest keys in last 5 minutes. It's only listing the top ops keys, not giving you a list of every key in the database and it's ops rating...

What's the use case for (2), maybe there is a better way of solving the use case?

Our buckets have too many items in them - I don't want to change the expiration time because I can't tell the difference between popular and unpopular content when I set it.

When you say too many items, are you saying you are having performance issues?

Regardless, you can still do this using your key strategy. So, basically if content isn't "popular" you want to delete it. You want to do this every 5 minutes?

Are you tracking hits on content? You can do that through key design, everything there is a request for a key you can also have a counter. Are you keying in a predictable way, in which case you can iterate over them at regular intervals and wipe them if they are lower than what you want?

Let me explain:

content::item_count
3

content::1
{...json...}

content::1::view_count
25

content::2
{...json...}

content::2::view_count
2

Everytime you request a content item, also send a incr(content_key + "::view_count). [I am not sure what language/sdk you are building with]

This is really general, you can take it further and have it based on how long it's been in the system (how new/old it is) and make it a rate of change, delta by adding a key based on a time signature of the last time it's changed, and calculate it's view-rate delta as a value.

content::1::20120718:1:55
25

content::1::20120718:2:00
35

content::1::20120717:2:00::view_delta
10

Simplified concept of course, I'd need to know more to give you more direct advice.
 

I will try to find a more direct answer for you, still working on it where you wouldn't necessarily need to use a key strategy.

One suggestion I have is use get and touch, so that if the TTL is hit, it deletes. You can set the TTL for 5 mins, or 24 hours, but that should give you the effect you desire.

I just read your thread here: http://www.couchbase.com/forums/thread/how-can-i-tell-couchbase-purge-ol...

which helps me understand why you are asking this...

How would I be able to query and remove keys that were not touched in x days?

I am not sure what your key structure is (how you are generating key names), so this can vary. If they keys are predictable, you can iterate through with a job with multi-gets, and if they are expired they will delete.

But if you are setting TTL, if the key is not touched within the TTL timeperiod, they will auto-expire. I will have to double check with engineering whether the service that deletes expired items runs in 1.8, in 2.0 I believe it runs once an hour, but I will confirm and re-reply to this thread tomorrow.

Yes, I verified, also in 1.8.x the service runs once an hour to delete TTL expired keys. Is this enough now to solve the design you are trying to accomplish?

Maybe I am going about this the wrong way.

How can I tune couchbase so that it evicts data from memory more frequently. For example, if my default couchbase waits for 2 days of inactivity before it move something from ram to disks - how can I set that value to 1 day. I think this solution will work best for me.

Thanks for all your help

Data is evicted from memory when Couchbase Server needs to free RAM to store more items in RAM. There is no explicit tuning necessary. How much RAM is used by the Bucket is controlled by the bucket quota. (more detail on the thresholds that will lead to memory being freed here: http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-bestpractic...). Guess should also mention that persistence of items, i.e. items being written to disk, is happening continuously. So freeing up memory focuses on just re-using memory used for items that have the current value persisted on disk, which is why this can happen fast and without major impact.

You are on the right track that you don't want to keep unused items around unless you have a good reason to do so, as there is Meta data (e.g. the keys) kept in memory.

So Scalabl3's strategy seems the best way. Use TTL and then GAT (Get And Touch) to "refresh TTL" on reads (and of course reset TTL on writes) to age out items that haven't been accessed.

Hope this helps.

Data is evicted from memory when Couchbase Server needs to free RAM to store more items in RAM. There is no explicit tuning necessary. How much RAM is used by the Bucket is controlled by the bucket quota. (more detail on the thresholds that will lead to memory being freed here: http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-bestpractic...). Guess should also mention that persistence of items, i.e. items being written to disk, is happening continuously. So freeing up memory focuses on just re-using memory used for items that have the current value persisted on disk, which is why this can happen fast and without major impact.

You are on the right track that you don't want to keep unused items around unless you have a good reason to do so, as there is Meta data (e.g. the keys) kept in memory.

So Scalabl3's strategy seems the best way. Use TTL and then GAT (Get And Touch) to "refresh TTL" on reads (and of course reset TTL on writes) to age out items that haven't been accessed.

Hope this helps.