Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.0-beta
-
Fix Version/s: 2.0
-
Component/s: view-engine
-
Security Level: Public
-
Labels:
-
Environment:1617
Centos 64 bit.
Description
1 bucket.
6 GB bucket, 8 gb system ram,
10M items
40% active and resident ratio.
It seems like that indexing 11 ddocs with 2 views each, seems to push the system into swap, pretty fast.
Seeing this on 2.0 large cluster, with similar setup as well.
What are the bounds for 2.0 when someone wants to use indexes and upto how many?
See the screenshot attached for what we are seeing, in the functional tests.
Views:-
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d1' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.key_num);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,doc.key);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d2' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.name);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.key_num]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d3' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.email);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.name]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d4' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.city);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.email]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d5' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.country);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.city]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d6' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.realm);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.country]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d7' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.coins);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.realm]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d8' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.category);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.coins]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d9' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.achievements);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.category]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d10' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.body);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.achievements]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d11' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.city+doc.country);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.body]);\n}"} }}'
6 GB bucket, 8 gb system ram,
10M items
40% active and resident ratio.
It seems like that indexing 11 ddocs with 2 views each, seems to push the system into swap, pretty fast.
Seeing this on 2.0 large cluster, with similar setup as well.
What are the bounds for 2.0 when someone wants to use indexes and upto how many?
See the screenshot attached for what we are seeing, in the functional tests.
Views:-
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d1' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.key_num);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,doc.key);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d2' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.name);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.key_num]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d3' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.email);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.name]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d4' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.city);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.email]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d5' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.country);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.city]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d6' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.realm);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.country]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d7' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.coins);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.realm]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d8' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.category);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.coins]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d9' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.achievements);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.category]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d10' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.body);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.achievements]);\n}"} }}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d11' \
-d '{"views": { "v1":{"map":"function(doc){\nemit(doc.key,doc.city+doc.country);\n}"}, "v2":{"map":"function(doc,meta){\nemit(meta.id,[doc.key_num,doc.body]);\n}"} }}'
Even more than that (assuming I'm not insane) is O(N^2) external sorting algorithm that we accidently implemented in couch_set_view_updater. Basically it keeps up to 16 1 meg files, then merges them to 1. Then continues with creating 15 1 meg files and merges everything together again. Then again, etc... It can be seen that this is quadratic behavior.