[MB-5938] UI needs to deal with very slow docloader (alternatively, docloader is too slow) Created: 18/Jul/12  Updated: 19/Feb/14  Resolved: 17/Sep/12

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: None
Fix Version/s: 2.0-beta-2, 2.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Pavel Paulau Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 1
Labels: 2.0-beta-release-notes
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

It works fine with old samples (gamesim-sample), but recent archives (beer-sample) contains much more data and usually fail due to timeout.

Now it takes at least 40 seconds to load documents and create views.

Comment by Aleksey Kondratenko [ 18/Jul/12 ]
Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don't think beers data is mega docs.
Comment by Pavel Paulau [ 18/Jul/12 ]
1. before loading these docs we have to create new bucket (if it doesn't exist).
2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird.
3. we also create new ddocs/views and perform queries on new indices.

if there is any problem with increasing timeout, I can fix point #2.
Comment by Aleksey Kondratenko [ 18/Jul/12 ]
Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate.
Comment by Pavel Paulau [ 18/Jul/12 ]
ok, fixing #2 doesn't help sometimes.

python client have extremely low throughput (regardless environment). and this is behind cbdocloader.
thus we need ~15-20 seconds to load these sample items.

I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4.
Comment by Aleksey Kondratenko [ 18/Jul/12 ]
Trying longer timeout. That sucks. We don't have good feedback for users.
Comment by Dipti Borkar [ 19/Jul/12 ]
Can we just make the load asynchronous and log problems?
Comment by Thuan Nguyen [ 26/Jul/12 ]
Integrated in github-ns-server-2-0 #412 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/412/])
    MB-5938: bumped up docloader timeout to 60 seconds (Revision f6d28497ddb2270962b8ba73284da66237120be8)

     Result = SUCCESS
Aliaksey Artamonau :
Files :
* src/menelaus_web.erl
Comment by Aleksey Kondratenko [ 27/Jul/12 ]
Assigning to Pavel for UI-only attempt to fix
Comment by Pavel Blagodov [ 17/Aug/12 ]
I had tried to POST on http://localhost:9001/sampleBuckets/install and after 2.8min of pending I got error 400 from server with the following response
{"error":true,"type":"docloader_failed","reason":"There was an unexpected error."}

In this case SAMPLES_LOADING_TIMEOUT was 12000000.
Comment by Aleksey Kondratenko [ 17/Aug/12 ]
There's blocker level bug on docloader for that
Comment by Karan Kumar (Inactive) [ 17/Aug/12 ]
Also look at

for other docloader improvements.
Comment by Aleksey Kondratenko [ 11/Sep/12 ]
Not actually fixed.
Comment by Aleksey Kondratenko [ 11/Sep/12 ]
Sorry. Meant to assign to Pavel
Comment by Pavel Blagodov [ 17/Sep/12 ]
Comment by Aleksey Kondratenko [ 17/Sep/12 ]
I've uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner.
Comment by kzeller [ 26/Oct/12 ]
Sample data will now be asynchronously loaded. Loading sample data will be
executed as a background task then later reported in web console
upon completion.
Generated at Wed Jul 09 22:07:34 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.