Hi,
We are currently using Couchbase as a data store for our chat application. Each user of our system has the ability to start a single conversation with another user or a group conversation with many of them. Currently we’re using this “schema”:
conversations:1
{ "id":1, "type": "SINGLE", "users": [1,2], "last_update": 1123455 }
conversations:2
{ "id":2, "type": "GROUP", "users": [1,2,3], "last_update": 1123455 }
user:1:conversations
[{"id":1, "type": "SINGLE", "last_update": 1123455}, {"id": 2, "type": "GROUP", "last_update": 1123455}]
user:2:conversations
[{"id":1, "type": "SINGLE", "last_update": 1123455}, {"id": 2, "type": "GROUP", "last_update": 1123455}]
user:3:conversations
[{"id":2, "type": "GROUP", "last_update": 1123455}]
conversations:1:messages_index
[{"page":1, "first_id": 0, "last_id": 2, "last_date": 1123455}]
conversations:1:messages_index:1
[{"id": 0, "date": 1234}, {"id": 1, "date": 12346}, {"id": 2, "date": 1123455}]
conversations:1:messages:0
{"id": 0, "text": "Hello", "sender": 1, "date": 1234}
We need to do all the Indexing by hand because couchbase views are not consistent enough (if a user receives a new message push notification and opens the application, the message has to be there - I believe this is called “Read your own writes”). We need to query messages between given dates, so we need those indexes.
I think is obvious to say that supporting concurrency with this approach is a pain (we need to update a lot of documents every time a message is sent)
My question is: Are we using views wrong or there is a way to use views to get rid of all those by-hand indexes? I know that we can set the staleness of our queries, but this is a HUGE view and the last time I tested it was a performance killer.
In an ideal scenario we would love to have:
conversations:1
{ "id":1, "users": [1,2], "last_update": 1123455 }
conversations:1:messages:0
{"id": 0, "text": "Hello", "sender": 1, "date": 1234}
and link those two between a view, so we can query all the messages of the conversation 1 between a range of dates.
We’re using Couchbase Server 3.0
One of the use cases of Couchbase is said to be “Chat Messaging” and I know Viber uses it as well, so the fact that we need to do such a complicated thing to solve our problem gets me to think we’re using this thing wrong. Another idea is to use Redis to do the indexing, but using a single datastore solution will be preferred.
We expect to have a lot of Users and conversations/messages (we are one of the most played games on both iOS and Android in the US, South America and Europe).
PS: In this article, Cihan Biyikoglu says:
“Stale=false type queries are super useful. Imagine building a messaging app. If the message you sent does not appear in your “sent messages” folder (which is typically a view query), you may send it again! Or you save a new playlist in your mobile app, you go back to your playlists and if you don’t see that new playlist, you are confused! So stale-false is essential!”
In that case, if we use views, what setup should we use to achieve Full Consistency on an index which has millions of updates a day?
Thanks in advance!