[MB-5938] UI needs to deal with very slow docloader (alternatively, docloader is too slow) Created: 18/Jul/12 Updated: 26/Oct/12 Resolved: 17/Sep/12 |
|
| Status: | Resolved |
| Project: | Couchbase Server |
| Component/s: | ns_server |
| Affects Version/s: | None |
| Fix Version/s: | recent-builds-2.0, 2.0-beta-2 |
| Security Level: | Public |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Pavel Paulau | Assignee: | Aleksey Kondratenko |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | 2.0-beta-release-notes | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
It works fine with old samples (gamesim-sample), but recent archives (beer-sample) contains much more data and usually fail due to timeout.
Now it takes at least 40 seconds to load documents and create views. |
| Comments |
| Comment by Aleksey Kondratenko [ 18/Jul/12 ] |
| Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don't think beers data is mega docs. |
| Comment by Pavel Paulau [ 18/Jul/12 ] |
|
1. before loading these docs we have to create new bucket (if it doesn't exist).
2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird. 3. we also create new ddocs/views and perform queries on new indices. if there is any problem with increasing timeout, I can fix point #2. |
| Comment by Aleksey Kondratenko [ 18/Jul/12 ] |
|
Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate.
|
| Comment by Pavel Paulau [ 18/Jul/12 ] |
|
ok, fixing #2 doesn't help sometimes.
python client have extremely low throughput (regardless environment). and this is behind cbdocloader. thus we need ~15-20 seconds to load these sample items. I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4. |
| Comment by Aleksey Kondratenko [ 18/Jul/12 ] |
| Trying longer timeout. That sucks. We don't have good feedback for users. |
| Comment by Dipti Borkar [ 19/Jul/12 ] |
| Can we just make the load asynchronous and log problems? |
| Comment by Thuan Nguyen [ 26/Jul/12 ] |
|
Integrated in github-ns-server-2-0 #412 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/412/]) Result = SUCCESS Aliaksey Artamonau : Files : * src/menelaus_web.erl |
| Comment by Aleksey Kondratenko [ 27/Jul/12 ] |
| Assigning to Pavel for UI-only attempt to fix |
| Comment by Pavel Blagodov [ 17/Aug/12 ] |
|
I had tried to POST on http://localhost:9001/sampleBuckets/install and after 2.8min of pending I got error 400 from server with the following response
{"error":true,"type":"docloader_failed","reason":"There was an unexpected error."} In this case SAMPLES_LOADING_TIMEOUT was 12000000. |
| Comment by Aleksey Kondratenko [ 17/Aug/12 ] |
| There's blocker level bug on docloader for that |
| Comment by Karan Kumar [ 17/Aug/12 ] |
|
Also look at
http://www.couchbase.com/issues/browse/MB-6274 for other docloader improvements. |
| Comment by Aleksey Kondratenko [ 11/Sep/12 ] |
| Not actually fixed. |
| Comment by Aleksey Kondratenko [ 11/Sep/12 ] |
| Sorry. Meant to assign to Pavel |
| Comment by Pavel Blagodov [ 17/Sep/12 ] |
| http://review.couchbase.org/20834 |
| Comment by Aleksey Kondratenko [ 17/Sep/12 ] |
| I've uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner. |
| Comment by Karen Zeller [ 26/Oct/12 ] |
|
Sample data will now be asynchronously loaded. Loading sample data will be
executed as a background task then later reported in web console upon completion. |