Couchbase Server

Initial indexing of 20M items is too long when view compaction is enabled

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Duplicate
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: view-engine
  • Security Level: Public
  • Labels:
  • Environment:
    4 nodes (80GB total ram quota, dedicated disk, dedicated network, CentOS 6.2)

Description

Steps:
1. Insert 20M items, 2K average items size, pretty random values.
2. Create 1 design doc with general 8 views.
3. Set 10% view fragmentation threshold.
4. Initialize indexing.

Looks like full indexing will take at least a day.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Pavel Paulau added a comment -
@Filipe
I'm pretty sure that you are aware about this issue and it doesn't sound surprising to you.
Just make it public and tracked.
Show
Pavel Paulau added a comment - @Filipe I'm pretty sure that you are aware about this issue and it doesn't sound surprising to you. Just make it public and tracked.
Hide
Permalink
Aleksey Kondratenko added a comment -
It was agreed that ns_server's autocompaction daemon should not trigger compaction of index that's still being built. But I'm not sure there's a way to distinguish index update and initial index building.
Show
Aleksey Kondratenko added a comment - It was agreed that ns_server's autocompaction daemon should not trigger compaction of index that's still being built. But I'm not sure there's a way to distinguish index update and initial index building.
Hide
Permalink
Filipe Manana added a comment -
Yes, the rule in the scheduler changed by Sharon's request.
There's currently no way to distinguish between initial/non-initial build/update.

For evperf tests, what you can do at the moment is set compaction fragmentation to 100% and after initial build finishes set it back to 10%.
I think it's worth trying to how better it gets for that evperf test, before doing more code changes.
Do you think it's easy to do in testrunner/evperf?
Show
Filipe Manana added a comment - Yes, the rule in the scheduler changed by Sharon's request. There's currently no way to distinguish between initial/non-initial build/update. For evperf tests, what you can do at the moment is set compaction fragmentation to 100% and after initial build finishes set it back to 10%. I think it's worth trying to how better it gets for that evperf test, before doing more code changes. Do you think it's easy to do in testrunner/evperf?
Hide
Permalink
Pavel Paulau added a comment -
For me it's easy, at least I have enough disk space.

But it still can be a problem for customers.
Show
Pavel Paulau added a comment - For me it's easy, at least I have enough disk space. But it still can be a problem for customers.
Hide
Permalink
Peter Wansch added a comment -
No longer applicable when CBD-422 is integrated.
Show
Peter Wansch added a comment - No longer applicable when CBD-422 is integrated.
Hide
Permalink
Pavel Paulau added a comment -
I'm not sure that we should close this issue.

9 hours is probably fine for 20M items, but what about 30M? 40M? It's still extremely long.
Show
Pavel Paulau added a comment - I'm not sure that we should close this issue. 9 hours is probably fine for 20M items, but what about 30M? 40M? It's still extremely long.
Hide
Permalink
Pavel Paulau added a comment -
Oh, resolution is duplicate. Than it's ok.
Show
Pavel Paulau added a comment - Oh, resolution is duplicate. Than it's ok.

People

  • Assignee:
    Filipe Manana
    Reporter:
    Pavel Paulau
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: