How can I force the compaction / compression of FTS index files using Scorch ?
I have detected that, in occasions, FTS generate index files of 20GB!!! that occupy 1000 times more than the others . They fill 100% my disk couchbase/data/@fts/fts_search_2664cb0ebe4623c4_54820232.pindex/store/000000001234.zap
In Moss, I have never had these problems (I am testing Scorch to reduce the size on disk) and I know that in Moss it is possible to define the % of fragmentation to which they want to trigger the Compaction
In Moss there is the tool mossScope that allows to know the fragmentation of an index: /opt/couchbase/bin/mossScope stats fragmentation path/to/myStore https://github.com/couchbase/mossScope
Is there a similar command in Scorch?
Iâve reviewed /logs/fts.log and total_compactions is always =0
Where can I see the fragmentation of my Scorch indexes?
I have detected that, when the FTS server is heavily loaded creating several FTS index at the same time, large .zap files are generate an never reduce o compress
If I create the indices fts 1 to 1 it doesnât usually happen (but sometimes, randomly, generate a large file).
If I delete and recreate the index, no large files are generates sometimes Did I find a bug?
Example:
couchbase@cb1:/opt/couchbase/var/lib/couchbase/data/@fts$ ls -la fts_search_2664cb0ebe4623c4_54820232.pindex/store/
total 20114336
drwx------ 2 couchbase couchbase 20480 Nov 7 20:58 .
drwx------ 3 couchbase couchbase 4096 Nov 7 11:49 âŚ
-rw------- 1 couchbase couchbase 11601527 Nov 7 12:07 000000007118.zap
-rw------- 1 couchbase couchbase 232229 Nov 7 12:07 00000000713d.zap
-rw------- 1 couchbase couchbase 16225914 Nov 7 12:07 000000007141.zap
-rw------- 1 couchbase couchbase 26941072 Nov 7 12:07 000000007192.zap
-rw------- 1 couchbase couchbase 20495543961 Nov 7 12:13 000000007205.zap <------------- 20GB !!!
-rw------- 1 couchbase couchbase 1242199 Nov 7 12:07 00000000724c.zap
-rw------- 1 couchbase couchbase 2347348 Nov 7 12:07 000000007448.zap
-rw------- 1 couchbase couchbase 1645901 Nov 7 12:07 0000000074f5.zap
-rw------- 1 couchbase couchbase 1086131 Nov 7 12:07 000000007684.zap
-rw------- 1 couchbase couchbase 1809879 Nov 7 12:07 0000000078df.zap
-rw------- 1 couchbase couchbase 1789122 Nov 7 12:07 0000000079b4.zap
-rw------- 1 couchbase couchbase 1655227 Nov 7 12:07 000000007a6a.zap
-rw------- 1 couchbase couchbase 2179977 Nov 7 12:07 000000007b32.zap
-rw------- 1 couchbase couchbase 1819414 Nov 7 12:07 000000007bdd.zap
-rw------- 1 couchbase couchbase 1848942 Nov 7 12:07 000000007c84.zap
-rw------- 1 couchbase couchbase 1718391 Nov 7 12:07 000000007d33.zap
-rw------- 1 couchbase couchbase 911983 Nov 7 12:13 000000007dd5.zap
-rw------- 1 couchbase couchbase 846549 Nov 7 12:13 000000007dd6.zap
-rw------- 1 couchbase couchbase 784477 Nov 7 12:13 000000007dd7.zap
-rw------- 1 couchbase couchbase 19457133 Nov 7 12:13 000000007dd8.zap
-rw------- 1 couchbase couchbase 665114 Nov 7 16:51 000000007f18.zap
-rw------- 1 couchbase couchbase 66874 Nov 7 17:55 00000000886b.zap
-rw------- 1 couchbase couchbase 809 Nov 7 17:56 00000000887e.zap
-rw------- 1 couchbase couchbase 228 Nov 7 17:56 000000008880.zap
-rw------- 1 couchbase couchbase 236 Nov 7 17:57 000000008882.zap
-rw------- 1 couchbase couchbase 236 Nov 7 17:57 000000008884.zap
-rw------- 1 couchbase couchbase 218 Nov 7 20:56 00000000895a.zap
-rw------- 1 couchbase couchbase 218 Nov 7 20:58 00000000895c.zap
-rw------- 1 couchbase couchbase 8388608 Nov 7 20:58 root.bolt
Is a not production enviroment, is a testing enviroment with no ops in CouchBase 6.0
Firstly, regarding total_compactions - it is a moss only stat, reason why it shows up as 0 with scorch.
Looking at all the zap files within fts_search_2664cb0ebe4623c4_54820232.pindex/store, it seems like there are many files that are rather old compared to the latest:
-rw------- 1 couchbase couchbase 218 Nov 7 20:56 00000000895a.zap
-rw------- 1 couchbase couchbase 218 Nov 7 20:58 00000000895c.zap
This could mean one of the following things:
Searchers are still holding on to older snapshots, but you mention that there arenât any other ops - so I think itâs safe to rule this one out.
We know of a bug which we have a fix for already but couldnât make it into 6.0.0 (but will definitely for a 6.0.1), where the scorch merger (which essentially is responsible for the compactions) doesnât run aggressively enough especially in situations where thereâs a lot of data, the fix for that is this for your reference: https://github.com/blevesearch/bleve/pull/1023
As for debugging this kind of a problem where one wantâs to track if compaction (file merging) with scorch isnât running often enough or is lagging, here are some stats that can assist you ⌠check out CurRootEpoch and LastMergedEpoch. If the LastMergedEpoch is very much behind the CurRootEpoch that should give you a clear indication that compaction has fallen behind and is perhaps even struggling to keep up.
Weâve noted an identical issue while testing in house and I can confirm that the above mentioned fix addresses the problem. Also pinging @steve and @sreeks if thereâs anything else that theyâd like to add.
Unlike moss, scorch will have many index files (zap) on disk at any given moment. And just a few random number of files wonât give us any clues on the progress/status of the compactions. And itâs totally expected that the index files are of varying sizes.
In any case, (single or multiple indexes) , if you end up observing overall bigger index file size on disk compared to that of moss, please do raise a bug report as it is never supposed to happen.
Scorch is supposed to bring you atleast 60% size reduction on disk as against moss/upside down indexes.
(It is a testing platform where we are evaluating the performance of Couchbase as a replacement for SolR. In Forestdb and Moss the disk space consumption and performance was worse than SolR, but in Scorch it improves a lot, good job! )
Iâd still love to help you with whatever you need. Donât hesitate to ask me any test
Indeed, I feel I have empirically proven that the anomalous growth of Scorch index files (due to no-merge/non-compaction) occurs in situations of high load or many data: for example creating many FTS indexes simultaneously (when doing a restore of all indexes) or when an index has many data (in my case between 2 and 10 million elements). In small indexes of less than 0.5 million items does not occur.
In our case, load or memory pressure should not be a problem because we have 3 nodes exclusively for indexes with 20GB RAM and 8 idle CPU, and no ops.
I suspect the bug is because of the number of items.
Attached is a graph of the size of the indexes:
CB 5.5.2 Moss = 400GB
CB 6.0 Scorch > 1000GB (Full Disk while creating 12 FTS Indexs concurrently)
Two Questions:
1.- For these files that have been created with anomalous size because have not been in-memory segment merging before persisting segments to disk. Would it be possible to manually compact the files on disk?
2.- Should the size of each pindex and its replica be similar in each server when everything works correctly?
Example: the pindex of âfts_notfoundâ are the same size os each replicas on each server, but in âfts_searchâ there is a lot of difference size between the pindex replicas.
Few points to keep in mind during these experiments,
Your finding about a single scorch index size looks correct and as per the expectation. (200GB size as compared to 450GB in upside down)
Not clear on the worry part of >1000 GB disk size with 12 scorch indexes. Indexes are not sharing any data on disk even if they are created on the same bucket. So I am not yet clear on why 1000 GB is too high for 12 indexes? Can you please help me understand this.
Your âCurRootEpoch = LastMergedEpochâ finding looks correct. If we roughly take the number of items - âTotIntroducedItemsâ stats value - 6158652 / ~6M is expected to create 32 files on disk and the ânum_files_on_diskâ: 32 stats confirms that. There are no wrinkles seen here.
The merge policy is expected to create a growing logarithmic staircase of segments, and hence its perfectly valid to have a few really large segment files and a higher number of smaller segment files on disk. We can further optimize the merge policy over the index creation/update operations. But will do it later as we are not yet fully clear about the issue here. And its also possible to have an older smaller segment file which might have outlived itâs peers during those compaction cycles.
The number of index / zap files are primarily controlled by the number of documents in it, and they are not created, split or merged based on their size to a larger extent.
Can you let us know what was the FTS memory quota set in these trials? In case of memory pressure situation as against the quota set, we would have the above mentioned issue of skipping the memory merge of segments to result in too many files on disk. And that could be a contributing factor to the " very different size" observed between a primary and the corresponding replica partition across nodes. Please share the stats for both of those partitions. May give some clues.
Trying to answer the original concerns raised above.
1.- For these files that have been created with anomalous size because have not been in-memory segment merging before persisting segments to disk. Would it be possible to manually compact the files on disk?
Nope, manual disk compaction isnât possible. If there arenât much updates or deletes to data, then compactions may not give you much size savings. Of course, its not a desirable situation as we will have several overheads from too many segment files on disk.
2.- Should the size of each pindex and its replica be similar in each server when everything works correctly?
Example: the pindex of âfts_notfoundâ are the same size os each replicas on each server, but in âfts_searchâ there is a lot of difference size between the pindex replicas.
With scorch index partitions (primary & replica), once the merger caught up with the root epoch, it is expected to be very close in size across primary and replica partitions.
Can you please cross check and share those two pindex stats similar to the one you shared already. Might have some clues there.
Yes. Scorch is a big improvement over Moss Great job
The size is relative
I currently use SolR/Lucent and the size of the indexes is 10 times smaller.
It is not an economic concern but a technical one.
We are a small company for which quality disk storage is the most expensive part of our hosting bill.
3 nodes exclusively dedicated to FTS with 20GB RAM each (13GB for FTS/node service)
I want to think that RAM is enough (itâs a test/development/trial environment)
In my test environment there are few index mutations by the few write ops.
Iâm worried that in a real environment with many mutations (like the ones that occur when you create the FTS index for the first time) If the compactions do not occur, there may be large orphan files that can fill my disk again.
What statistics exactly do you need? stats:? managerStats:? bleveIndexStats?
Should I get them from the fts.logs or is there a better method?
We do not offer a way to manually compact files on disk.
The size of a pindex and replica are expected to be somewhat similar but that depends on if and when the mergerâs run on the zap files in each of the pindexes.
The bug that I mentioned earlier - where the merger lags, isnât a consistently seen issue. I believe youâd see more uniform sizes with the fix that is going to make it into 6.0.1.
All the pindex stats are dumped into the log files periodically, so Iâve a few questions on the stat snapshot you have highlighted above:
Does it belong to the index whose zap files show the huge difference in the sizes?
Are you looking at the latest snapshot of the stats?
If youâd be willing - we could take a look at the logs for you. Youâll need to run the cbcollect_info command either on the logs collection tab on the UI or from a CLI, and share these logs.zip here.
I forgot to hit send on my reply^, and looks like @sreeks had already answered youâre previous questions
@treo Weâll be able to better assist you if we could look into the logs as well. Iâve shared the instructions on how to collect them in my previous message
Agreed!
Before glancing into any of your logs , another sizing related aspect coming to mind is about the index type mappings you might have used. Please try to index/ provide a mapping for the fields which you need to be searchable and make sure to turn off the default mapping in the index definition as this would have undesirable effect on index size.
Thank you so much for the advice @sreeks .
I have reviewed it: all our FTS indexes only mapping fields which we need to be searchable/storable.
We allways disable âinclude_term_vectorsâ in all text field
I allways turn off the default mapping and any dinamic fields.
Example:
meta().id: "photos::10000001"
{
"pho_id": 10000001,
"pho_file": "80a4e55bf8798647e26234ab2e8da8af1.jpg",
"pho_sou_id": 5,
"pho_title": "Nec pretium nunc lorem. Elit non, diam massa, neque consectetuer.",
"pho_description": "FILE PHOTO: TLorem ipsum dolor sit amet, et vulputate sapien libero leo quis, metus ipsum ut nunc, tellus amet at aenean maecenas tempor, venenatis lorem, phasellus semper lacinia nunc eleifend ultrices. Mi consectetuer, tempor cursus, lacinia elit ultricies. Proin sit amet nonummy vestibulum vehicula. Nec pretium nunc lorem. Elit non, diam massa, neque consectetuer. Switzerland May 18, 2018. Picture taken May 18, 2018. AGENT Name Surname File Photo",
"pho_keywords": "",
"pho_creation_date": "2018-05-24 14:40:48",
"pho_author": "Name Surname",
"pho_original_name": "2018-05-24T124048Z_562818715_RC15C86AD870_RTRMADP_3_REATSE_PHOTO.JPG",
"pho_insert_date": "2018-05-24 14:41:13",
"entitytype": "photos"
}
One last question:
How could I check that there are no fields in the index that I donât want to index?
Iâm trying the âcbft-bleveâ cli command but it doesnât return information
data/@fts$ /opt/couchbase/bin/cbft-bleve scorch info fts_places_27dc801239cb56d7_13aa53f3.pindex/store/
Give a try with â/opt/couchbase/bin/cbft-bleve fields
ns_server/data/n_0/data/@fts/FTS_PARTITION_PATH.pindex/â should help with the fields indexed.
Not sure of the issue here, the first command works for me. Will check that. @abhinav any clues here?
Another potential size contributing factor in scorch indexes compared to that of solr/lucene could be that for the docValues. Scorch always stores the docValues and there is no configurability around this.
We have noted this and will make it configurable in coming releases.
Can you confirm whether your solr indexes have docValues enabled or disabled?
@sreeks , I confirm that in SolR/Lucene we are not using docValues (we do not use function queries or faceting). Could this be the reason why Lucene indexes occupy 10 times less than Scorch?
@treo, this could be one contributing factor. But that canât account for a 10X amplification. Lucene has optimised storage techniques further in many aspects, for eg: numeric fields.
Just to be clear on the exact size difference you observe, the 200GB scorch index on SolR becomes around 20GB?
The sum of the 12 indexes (cores) in SolR occupy 20GB on disk (1 single server).
The same 12 indexes in Scorch occupy 200GB (double, 400GB in Moss). Evidently double if we add 1 replicas (400GB)
We use numerical fields in SolR (and no DocValues), this could be a reason for space optimization in SolR, but I think it is not the main reason for space amplification because there are indices/cores where we do not use numerics.
2 examples of comparing size and index definition: SolR vs Scorch (Same indexed fields, no dinamic files, no defauld_mapping) :
Example 1.- Photo index. 4,6GB in SolR vs 25GB in Scorch
SolR Sizes:
ls -lh photo/data/index/
total 4,6G <---------------------------------------
2,1G nov 6 03:12 _4h1k2.fdt
1,2M nov 6 03:12 _4h1k2.fdx
843 nov 6 03:15 _4h1k2.fnm
866M nov 6 03:15 _4h1k2_Lucene41_0.doc
448M nov 6 03:15 _4h1k2_Lucene41_0.pos
1,1G nov 6 03:15 _4h1k2_Lucene41_0.tim
13M nov 6 03:15 _4h1k2_Lucene41_0.tip
62M nov 6 03:15 _4h1k2.nvd
101 nov 6 03:15 _4h1k2.nvm
447 nov 6 03:15 _4h1k2.si
102 nov 11 03:10 segments_44umt
20 nov 11 03:10 segments.gen
0 jul 19 02:38 write.lock
If your requirement is limited to field scoped queries, then you may untick the " include in _all field" option for the fields indexed. This could save some space as well.
We havenât yet benchmarked scorch sizing with respect to that of lucene, so may be your findings are hints for further improvements. Certainly this helps in prioritising the tasks at our end.
Hi â still scanning /scrolling through this long thread, and one thing popped out (not sure if it was already mentioned)âŚ
A lot of numeric fields.
I think I was seeing a lot of id fields being indexed as type ânumberâ in your sample index definition. Indexing as a number is useful if youâve got real numeric use cases happening, like range searches, like âsearch for all the product docs that are about âiphoneâ and whose price < 1000â
But many apps donât actually perform range searches on id fields but only do exact keyword equality searches on id fields, so I often find it better to instead use text type and keyword analyzer instead of ânumberâ for these kinds of id field situations, which can save on index size. (Again, if this was already mentioned somewhere on the thread, apologies!)
You are absolutely right!, many of our fields are ânumberâ because we search for this style:
âsearch for all the product docs that are about âiphoneâ and active=1 and category=25 and enable=1 and type=33â
Effectively we donât usually do numerical searches like category<25
We chose numerical fields because we have it in our DB model and because we thought it would be more efficient in FTS (Possibly I was wrong).
Now, I change some numeric fields to text field and ⌠disk index size reduced from 76GB to 40GB
Do you think it will also improve search performance?
Tomorrow I will test the performance and analyze my applications querys to see if I can change from numerical to text without affecting the application.
Do you have any document/blog/post of good practices about creating FTS indexes in Couchbase?
Thanks!!
Did I find a bug?
(Full Disk while creating 12 FTS Indexs concurrently)
Yes. Scorch is a big improvement over Moss 
