index performance
Tue, 05/15/2012 - 23:39
I can't tell what is going wrong, but indexing is taking days (> 48hrs, < 50% indexed) on some buckets with a small number of views and relatively small number of documents each (< 10million). The map function is essentially this:
if (/^type:.*/.test(doc.id_field)) emit (doc.id, doc)
There is a recommendation not to emit the doc, but I've encountered multiple issues trying to retrieve the document from the view so gave up one emitting null for the value. I had originally thought indexing was stalled, but now I think it is just very slow for some reason.
Just did an iostat check -- reads and writes are at about 1400 blocks/sec and 53% utilization. Is this high? How does Couchbase index documents?
Hello,
We are aware of non-optimal performance for index builds/updates and queries for many scenarios. For 2.0 DP4 we had a brand new view engine implementation, therefore most of the efforts went into making it work, be stable and as bug free as possible.
We have already many big performance improvements (from 2x up to 10x for some cases) for the next developer preview (DP5) and we're working on more big performance enhancements, always with stability and correctness in mind.
As a side note, the view you're creating likely matches a worst case scenario - if your documents id field is an UUID. However what you're reporting, > 48 hours for less than 50%, is definitely excessive. What kind of hardware is it and how many nodes?
best regards,
Filipe Manana