What kind of strategy can you recommend for game development?
Hi, everyone.
Now I select nosql database for developing of social game. I know that Couchbase used by Zynga and other big company. And I can't understand some things with Couchbase.
Couchbase stores all keys in memory. So if we take any game like Farm we can see that one user has a lot of keys in game :
- key for user itself,
- few keys for areas (new territories),
- a lot of keys for plants, buildings, fields and other game units.
In general, a single user has an average of around 1000 keys. 1000 keys with 20 bytes length + 160 bytes for meta data ~ 160KB per user in memory without any documents. 100 000 users take about 16GB in memory only for keys, i.e. 100 000 users per one physic server with 24GB RAM (16GB for keys and 8GB for real data and anything else). It's very expensive solution. Gamer can install game and leave it, but keys will be in memory forever. I can't remove his keys, because I don't know when gamer will return.
In MongoDB I can use one big document which contains all game units for user and update it partial. But Couchbase hasn't support partial update. If I'll keep all game units in one or few big documents in Couchbase, I'll spend a lot of traffic.
What kind of strategy can you recommend for game development? Couchbase has a lot of sweet features. I would like to use them.
Welcome any advice and recommendations.
Yes, right. Each node keeps that vbucket keys + metadata, but anyway one node can contains 100000 - 200000 users with 1000 keys per user (100 000 000 keys per node).
Let's say I can combine all of these game units into a single document. For example (each field or decor has more properties then in the example):
{ u436254_a1_field:{ 1:{x:10,y:20,status:1,plant:sunflower,left:300}, 2:{x:14,y:24,status:1,plant:bamboo,left:360}, ... 636:{x:104,y:98,status:1,plant:bamboo,left:60}, } } { u436254_a1_decor:{ 1:{x:40,y:60,typeID:10}, 2:{x:42,y:70,typeID:12}, ... 760:{x:120,y:50,typeID:8} } }
Will it be normal practice get all fields or decor in any time when user change one single property of some field or decoration?
As you've seen, it would not be a good idea to have 1000 keys per-user. Many applications have a few, but on the order of less than 10 for specific purposes.
With Couchbase, you really don't need to worry about reading/writing the whole key at once. Our caching layer provides extremely low latency and high throughput, and the even distribution across all the nodes in a cluster causes them all to participate in serving the data in parallel. Many other databases (Mongo included) don't have these features and so suffer much more from bottlenecks and slow downs over a higher workload.
The only times you really need to worry about splitting a document up is a) if the data within is going to grow without bounds (like a comment list on a blog) or b) if multiple processes are going to try to change different pieces of the data very constantly in which case you would have a lot of contention on that single document. In these cases, it's common to "refer" to another document from within the main users' document (for example) and then you can update that/those documents separately. As I mentioned above, you don't want to go overboard on this and in most cases it's not necessary.
Regarding the memory, it is true that we keep all the keys and metadata (but not necessarily the values) in RAM at all times. This provides extremely fast lookup times, and even faster "miss" times for when data doesn't exist which is very important for some use cases. Over the next few versions we are looking explicitly at reducing the overhead per-key, but we currently have customers storing multiple billions of keys within a single cluster. It's simply a matter of understanding your performance and sizing requirements.
Hope that helps
Perry
The only times you really need to worry about splitting a document up is a) if the data within is going to grow without bounds (like a comment list on a blog)
Thanks Perry to your answer. I have last question. What happens if the documents will be constantly changing its size?
For example: user makes 500 plants (document with plants increased gradually from 1 to 500 items), 300 plants was grew and player removed them to warehouse (document with plants decreased gradually from 500 to 200 items). And this process will always be repeated.
Thanks to auto-compaction in 2.0, changing document sizes don't have a major performance impact. There is a potential for memory fragmentation, but that rarely is a issue in deployments (as some new docs will re-use the memory).
Couchbase only keeps the active set of documents in memory, not the entire database. It will page dormant documents to disk. But yes I think it does keep all keys in memory for that vbucket. But if you have multiple nodes, each node only keeps that vbucket keys + metadata, right?