Is there a bulk get operation in couchbase lite (similar to that which is in the sync gateway)
It’s not necessary in the native API because the library is embedded so there is no latency to sending requests.
In the REST API you can query _all_docs with a keys parameter listing the docIDs you want.
The keys parameter isn’t in the docs. And when I tried to use it I got this response:
{ status: 400,
error: ‘Invalid parameter in HTTP query or JSON body’ }
What should the url look like?
The keys parameter should be URL encoded (i.e. don’t put [ and stuff directly in) into the URL in the query portion. That might be where things are going wrong.
Sorry, yes just realised I missed the encoding step. Thanks.
I’m having some trouble with the _all_docs solution for bulk loading documents via the couchbase lite rest api.
When document is removed from a channel the _all_docs endpoint returns the document, with no indiction that it’s been removed.
The changes feed doesn’t indicate its been removed either, it just looks like this:
{
seq: 1856,
id: "-yimopGaRB2LFnzRvPULusN",
changes: [
{
rev: "3-23f1c5a1c32f8bdbf78f8ae22895639b"
}
]
}
If I get the document it comes back with _removed: true as expected:
{
_removed: true,
_id: "-yimopGaRB2LFnzRvPULusN",
_rev: "3-23f1c5a1c32f8bdbf78f8ae22895639b"
}
Is there an efficient way to bulk load documents that shows me removals? Getting docs individually is far too slow when there are very many to load. Or can I make the changes feed show that a document has been removed?
To be clear, I’m not talking about deleting documents. Rather document that have been removed from a channel.
FYI - the all_docs query returns this:
{
total_rows: 1,
offset: 0,
rows: [
{
{
value: {
rev: "3-23f1c5a1c32f8bdbf78f8ae22895639b"
},
key: "-yimopGaRB2LFnzRvPULusN",
id: "-yimopGaRB2LFnzRvPULusN",
doc: {
_id: "-yimopGaRB2LFnzRvPULusN",
_rev: "3-23f1c5a1c32f8bdbf78f8ae22895639b"
}
}
]
}I have the same issue when removing a document from a channel. Even if the removed channel has already been synced to Couchbase Server, the user that belongs to the channel still gets the document (old revision) when using
var query = database.CreateAllDocumentsQuery();
on client device.
See my post Document removed channel not replicated
Interesting. I’m not 100% sure we have exactly the same problem but it does sound related. I can’t now reproduce what I have above, but essentially when a document is removed from the channel I do not seem to get the removed flag in the changes feed. Unless of course I’m not actually removing it! I can’t find a clear answer of how to remove a document from a channel in the docs.
Hm. This problem’s not related to how you query the db — it sounds like your database simply doesn’t have the revision with the _removed flag. Which implies a problem on the Sync Gateway side — either you’re not actually removing the document, or there’s a bug in Sync Gateway.
(To remove a doc from a channel you simply don’t call channel() to add it to that channel, when it’s updated. In other words, Sync Gateway recomputes the set of channels the doc is in every time the doc is updated.)
It looks like you’ve got multiple threads going for this issue - I added a couple of followup questions on your other thread:
Jens, I think my local client database received the revision with the _removed flag.
Again my situation : User 1 adds User2 to channels property of its document. Later User2 will be removed from the document, but still gets the document (old revision) when querying its client database with the code in my last post. The document has the id -7hqnQ_TdLUe1jEr8_j0WaQ in the following log outputs.
User1 adds User2 to channels property:
2016-03-30T10:58:00.582+02:00 HTTP: #080: GET /hamster/
2016-03-30T10:58:00.582+02:00 HTTP: #080: → 401 Login required (0.7 ms)
2016-03-30T10:58:00.593+02:00 HTTP: #081: GET /hamster/
2016-03-30T10:58:00.593+02:00 HTTP: #081: → 401 Login required (0.6 ms)
2016-03-30T10:58:00.645+02:00 HTTP: #082: GET /hamster/_local/8008588c0780787edc7ff92cd27ed3186ae6a086 (as User1)
2016-03-30T10:58:00.667+02:00 HTTP: #083: GET /hamster/_local/2dbdb51045795b9512fab58424f80c86f665bbd3 (as User1)
2016-03-30T10:58:00.711+02:00 HTTP: #084: POST /hamster/_revs_diff (as User1)
2016-03-30T10:58:00.761+02:00 HTTP: #085: POST /hamster/_changes (as User1)
2016-03-30T10:58:00.764+02:00 Changes+: Int sequence multi changes feed…
2016-03-30T10:58:00.764+02:00 Changes: MultiChangesFeed({User1}, {Since:11 Limit:0 Conflicts:true IncludeDocs:false Wait:false Continuous:false Terminator:0xc8204e66c0 HeartbeatMs:300000 TimeoutMs:300000 ActiveOnly:false}) … (to User1)
2016-03-30T10:58:00.764+02:00 Changes+: MultiChangesFeed: channels expand to channels.TimedSet{“User1”:channels.VbSequence{VbNo:(uint16)(nil), Sequence:0x2}} … (to User1)
2016-03-30T10:58:00.764+02:00 Changes+: Sending seq:12 from channel User1
2016-03-30T10:58:00.764+02:00 Changes+: MultiChangesFeed sending &{Seq:12 ID:-7hqnQ_TdLUe1jEr8_j0WaQ Deleted:false Removed:{} Doc:map[] Changes:[map[rev:2-3ab7ef686d5bbde3add5295e8185a6c4]] Err: allRemoved:false branched:false} (to User1)
2016-03-30T10:58:00.765+02:00 Changes: MultiChangesFeed done (to User1)
2016-03-30T10:58:00.870+02:00 HTTP: #086: POST /hamster/_bulk_docs (as User1)
2016-03-30T10:58:00.876+02:00 CRUD+: Invoking sync on doc “-7hqnQ_TdLUe1jEr8_j0WaQ” rev 3-5b9c8c62d01ec95308f36f30939c0aa9
2016-03-30T10:58:00.876+02:00 CRUD+: Saving old revision “-7hqnQ_TdLUe1jEr8_j0WaQ” / “2-3ab7ef686d5bbde3add5295e8185a6c4” (462 bytes)
2016-03-30T10:58:00.876+02:00 CRUD+: Backed up obsolete rev “-7hqnQ_TdLUe1jEr8_j0WaQ”/“2-3ab7ef686d5bbde3add5295e8185a6c4”
2016-03-30T10:58:00.876+02:00 CRUD: Doc “-7hqnQ_TdLUe1jEr8_j0WaQ” in channels “{User1, User2}”
2016-03-30T10:58:00.878+02:00 CRUD: Stored doc “-7hqnQ_TdLUe1jEr8_j0WaQ” / “3-5b9c8c62d01ec95308f36f30939c0aa9”
2016-03-30T10:58:00.883+02:00 HTTP: #087: PUT /hamster/_local/2dbdb51045795b9512fab58424f80c86f665bbd3 (as User1)
2016-03-30T10:58:01.006+02:00 HTTP: #088: PUT /hamster/_local/8008588c0780787edc7ff92cd27ed3186ae6a086 (as User1)
2016-03-30T10:58:01.869+02:00 Changes+: Notifying that “hamster” changed (keys="{, User1, User2}") count=8
User2 sync and gets the document when querying local database:
2016-03-30T10:58:41.993+02:00 HTTP: #089: GET /hamster/
2016-03-30T10:58:41.993+02:00 HTTP: #089: → 401 Login required (1.0 ms)
2016-03-30T10:58:41.997+02:00 HTTP: #090: GET /hamster/
2016-03-30T10:58:41.997+02:00 HTTP: #090: → 401 Login required (0.3 ms)
2016-03-30T10:58:42.060+02:00 HTTP: #091: POST /hamster/_changes (as User2)
2016-03-30T10:58:42.064+02:00 Changes+: Int sequence multi changes feed…
2016-03-30T10:58:42.064+02:00 Changes: MultiChangesFeed({User2}, {Since:0 Limit:0 Conflicts:true IncludeDocs:false Wait:false Continuous:false Terminator:0xc8204e7260 HeartbeatMs:300000 TimeoutMs:300000 ActiveOnly:false}) … (to User2)
2016-03-30T10:58:42.064+02:00 Changes+: MultiChangesFeed: channels expand to channels.TimedSet{“User2”:channels.VbSequence{VbNo:(*uint16)(nil), Sequence:0x3}} … (to User2)
2016-03-30T10:58:42.064+02:00 Changes+: Sending seq:8 from channel User2
2016-03-30T10:58:42.064+02:00 Changes+: Sending seq:13 from channel User2
2016-03-30T10:58:42.064+02:00 Changes+: MultiChangesFeed sending &{Seq:3 ID:_user/User2 Deleted:false Removed:{} Doc:map Changes: Err: allRemoved:false branched:false} (to User2)
2016-03-30T10:58:42.065+02:00 Changes+: MultiChangesFeed sending &{Seq:8 ID:-Qkq42cILQEOyH8hbv7g5YQ Deleted:false Removed:{} Doc:map Changes:[map[rev:1-01671a1a65cd4756526258fb1514aa12]] Err: allRemoved:false branched:false} (to User2)
2016-03-30T10:58:42.065+02:00 Changes+: MultiChangesFeed sending &{Seq:13 ID:-7hqnQ_TdLUe1jEr8_j0WaQ Deleted:false Removed:{} Doc:map Changes:[map[rev:3-5b9c8c62d01ec95308f36f30939c0aa9]] Err: allRemoved:false branched:false} (to User2)
2016-03-30T10:58:42.065+02:00 Changes: MultiChangesFeed done (to User2)
2016-03-30T10:58:42.068+02:00 HTTP: #092: GET /hamster/_local/f3a1df14a5517abf735a23da011213fa4cfe3d27 (as User2)
2016-03-30T10:58:42.160+02:00 HTTP: #093: PUT /hamster/_local/f3a1df14a5517abf735a23da011213fa4cfe3d27 (as User2)
2016-03-30T10:58:42.161+02:00 HTTP: #094: GET /hamster/-7hqnQ_TdLUe1jEr8_j0WaQ?rev=3-5b9c8c62d01ec95308f36f30939c0aa9&revs=true&attachments=true (as User2)
2016-03-30T10:58:43.699+02:00 HTTP: #095: PUT /hamster/_local/66c75786a96286ff2d653aeac48eb2746440cb77 (as User2)
User1 removed User2 from channel:
2016-03-30T11:00:04.495+02:00 HTTP: #096: GET /hamster/
2016-03-30T11:00:04.496+02:00 HTTP: #096: → 401 Login required (0.6 ms)
2016-03-30T11:00:04.504+02:00 HTTP: #097: GET /hamster/
2016-03-30T11:00:04.504+02:00 HTTP: #097: → 401 Login required (0.5 ms)
2016-03-30T11:00:04.564+02:00 HTTP: #098: GET /hamster/_local/8008588c0780787edc7ff92cd27ed3186ae6a086 (as User1)
2016-03-30T11:00:04.566+02:00 HTTP: #099: GET /hamster/_local/2dbdb51045795b9512fab58424f80c86f665bbd3 (as User1)
2016-03-30T11:00:04.646+02:00 HTTP: #100: POST /hamster/_revs_diff (as User1)
2016-03-30T11:00:04.675+02:00 HTTP: #101: POST /hamster/_changes (as User1)
2016-03-30T11:00:04.675+02:00 Changes+: Int sequence multi changes feed…
2016-03-30T11:00:04.675+02:00 Changes: MultiChangesFeed({User1}, {Since:12 Limit:0 Conflicts:true IncludeDocs:false Wait:false Continuous:false Terminator:0xc8206649c0 HeartbeatMs:300000 TimeoutMs:300000 ActiveOnly:false}) … (to User1)
2016-03-30T11:00:04.676+02:00 Changes+: MultiChangesFeed: channels expand to channels.TimedSet{“User1”:channels.VbSequence{VbNo:(uint16)(nil), Sequence:0x2}} … (to User1)
2016-03-30T11:00:04.676+02:00 Changes+: Sending seq:13 from channel User1
2016-03-30T11:00:04.676+02:00 Changes+: MultiChangesFeed sending &{Seq:13 ID:-7hqnQ_TdLUe1jEr8_j0WaQ Deleted:false Removed:{} Doc:map[] Changes:[map[rev:3-5b9c8c62d01ec95308f36f30939c0aa9]] Err: allRemoved:false branched:false} (to User1)
2016-03-30T11:00:04.676+02:00 Changes: MultiChangesFeed done (to User1)
2016-03-30T11:00:04.721+02:00 HTTP: #102: POST /hamster/_bulk_docs (as User1)
2016-03-30T11:00:04.723+02:00 CRUD+: Invoking sync on doc “-7hqnQ_TdLUe1jEr8_j0WaQ” rev 4-8e85634c4905d9081b2730900e33f537
2016-03-30T11:00:04.723+02:00 CRUD+: Saving old revision “-7hqnQ_TdLUe1jEr8_j0WaQ” / “3-5b9c8c62d01ec95308f36f30939c0aa9” (517 bytes)
2016-03-30T11:00:04.724+02:00 CRUD+: Backed up obsolete rev “-7hqnQ_TdLUe1jEr8_j0WaQ”/“3-5b9c8c62d01ec95308f36f30939c0aa9”
2016-03-30T11:00:04.724+02:00 CRUD: Doc “-7hqnQ_TdLUe1jEr8_j0WaQ” in channels “{User1}”
2016-03-30T11:00:04.725+02:00 CRUD: Stored doc “-7hqnQ_TdLUe1jEr8_j0WaQ” / “4-8e85634c4905d9081b2730900e33f537”
2016-03-30T11:00:04.778+02:00 HTTP: #103: PUT /hamster/_local/2dbdb51045795b9512fab58424f80c86f665bbd3 (as User1)
2016-03-30T11:00:04.813+02:00 HTTP: #104: PUT /hamster/_local/8008588c0780787edc7ff92cd27ed3186ae6a086 (as User1)
2016-03-30T11:00:09.897+02:00 Changes+: Notifying that “hamster” changed (keys="{, User1, User2}") count=9
User2 sync and still gets the old revision of the document when querying its local database :
2016-03-30T11:00:37.429+02:00 HTTP: #105: GET /hamster/
2016-03-30T11:00:37.429+02:00 HTTP: #105: → 401 Login required (0.6 ms)
2016-03-30T11:00:37.434+02:00 HTTP: #106: GET /hamster/
2016-03-30T11:00:37.434+02:00 HTTP: #106: → 401 Login required (0.3 ms)
2016-03-30T11:00:37.493+02:00 HTTP: #107: GET /hamster/_local/f3a1df14a5517abf735a23da011213fa4cfe3d27 (as User2)
2016-03-30T11:00:37.502+02:00 HTTP: #108: GET /hamster/_local/66c75786a96286ff2d653aeac48eb2746440cb77 (as User2)
2016-03-30T11:00:37.557+02:00 HTTP: #109: POST /hamster/_revs_diff (as User2)
2016-03-30T11:00:37.571+02:00 HTTP: #110: POST /hamster/_changes (as User2)
2016-03-30T11:00:37.575+02:00 Changes+: Int sequence multi changes feed…
2016-03-30T11:00:37.575+02:00 Changes: MultiChangesFeed({User2}, {Since:13 Limit:0 Conflicts:true IncludeDocs:false Wait:false Continuous:false Terminator:0xc8204e7980 HeartbeatMs:300000 TimeoutMs:300000 ActiveOnly:false}) … (to User2)
2016-03-30T11:00:37.576+02:00 Changes+: MultiChangesFeed: channels expand to channels.TimedSet{“User2”:channels.VbSequence{VbNo:(*uint16)(nil), Sequence:0x3}} … (to User2)
2016-03-30T11:00:37.576+02:00 Changes+: Sending seq:14 from channel User2
2016-03-30T11:00:37.576+02:00 Changes+: MultiChangesFeed sending &{Seq:14 ID:-7hqnQ_TdLUe1jEr8_j0WaQ Deleted:false Removed:{User2} Doc:map Changes:[map[rev:4-8e85634c4905d9081b2730900e33f537]] Err: allRemoved:true branched:false} (to User2)
2016-03-30T11:00:37.576+02:00 Changes: MultiChangesFeed done (to User2)
2016-03-30T11:00:37.627+02:00 HTTP: #111: PUT /hamster/_local/f3a1df14a5517abf735a23da011213fa4cfe3d27 (as User2)
2016-03-30T11:00:37.666+02:00 HTTP: #112: PUT /hamster/_local/66c75786a96286ff2d653aeac48eb2746440cb77 (as User2)
Andreas - can you please confirm that you’re seeing the removed flag (not the deleted flag) when a document is removed from a channel?
Where do you see it? On in document itself, or in the _changes feed?
The document in my local database doesn’t have it. I can see the removal happening in the sync-gateway logs. And can see the flag in the sync-gateway _changes feed if I connect with the correct user, but I can see it anywhere i the local database.
I’m not quite sure if it’s correct log entry, but I think the following entry from SyncGateway output in my last post should indicate that the user should have been removed from the channel, actually:
2016-03-30T11:00:37.576+02:00 Changes+: MultiChangesFeed sending &{Seq:14 ID:-7hqnQ_TdLUe1jEr8_j0WaQ Deleted:false Removed:{User2} Doc:map Changes:[map[rev:4-8e85634c4905d9081b2730900e33f537]] Err: allRemoved:true branched:false} (to User2)
The following code on my client returns false (think this will check the removed flag on my local document?!):
Debug.WriteLine("DOC gone: " + doc.CurrentRevision.IsGone);
Also, the document still has the channels property with old values.
Hence, I think we got the same issue. SynGateway log shows removal, but local database / document doesn’t get updated.