Details
-
Type:
Improvement
-
Status:
Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.0
-
Fix Version/s: 2.0.1
-
Component/s: view-engine
-
Security Level: Public
-
Labels:None
Description
Use very large on disk sorted batches, to reduce update time and fragmentation.
Also benefit compaction retry phase.
More important during rebalance or when user has automatic index updates disabled, and each indexer run processes very large number of changes.
Also benefit compaction retry phase.
More important during rebalance or when user has automatic index updates disabled, and each indexer run processes very large number of changes.
From commit message, some benchmarks:
"
Test 1:
Initially 3.2M documents, design document with 4 views (both
from evperf lucky8 tests), index was built and later more
3.2M documents were added (new 512 partitions added to the
index as active).
Initial build took: 11m17.911s
Index file size after initial build: 380Mb
** Before this change
Incremental update took: 75m43.575s
Index file size after incremental update: 30.0Gb
** After this change
Incremental update took: 18m36.762s
Index file size after incremental update: 4.0Gb
Test 2:
Initially 3.2M documents (from lucky8 evperf test), design
document with a single view (emits meta.id key and null value
for each document), index was built and later more 3.2M documents
were added (new 512 partitions added to the index as active).
Initial build took: 6m23.199s
Index file size after initial build: 134Mb
** Before this change
Incremental update took: 21m31.615s
Index file size after incremental update: 8.6Gb
** After this change
Incremental update took: 7m13.394s
Index file size after incremental update: 1.8Gb
"