I have created two cluster nodes (db1 and db2) running CentOS 7.6, Couchbase Community Server Edition 6.0.0 build 1693
Through code I create the same indexes on both clusters (using the Java SDK). However, I have now discovered that there are some major differences on the two servers’ use of disk.
The database takes up 2.15 GB on db1 and 2.23 GB on db2.
However, in the directory: opt/couchbase/var/lib/couchbase/data there are some major differences. On db2 this directory has a size of 4.11 KB whereas on db1 it takes up 2.34 GB!!
On db2 there is only one directory in data which is called .delete - and it is empty. On db1 there are several directories in addition to that one, namely: @2i (767 MB), @fts (32 bytes), @indexes (66 MB), and data (1.53 GB).
In the data directory there are many files named like: 0.couch.16, 1.couch.15, 10.couch.13, etc.
So this leads me to a couple of questions:
Why is there such a difference on two nodes that are clustered?
Why do the @-directories not show up on both servers?
What is the data in the lowest level data directory which makes the biggest difference?
Sometimes I see the db2 server get quite unresponsive (like not being able to open the Couchbase console - or even Webmin or SSH to the server). Could there be any connection to the above differences? A reboot normally resolves the issue (or rather clears the symptom)
The above data are in a test environment - so no stress here. However, I have a similar production environment (without the size differences) - and therefore I’m quite curious to understand what’s going on in the test environment so I can be prepared if something like this should hit the production environment…
Those directories are important e.g. @2i would hold all the GSI indexes created via query service. It is surprising that on db2 you were able to create indexes and there is no @2i directory. You may want to search that directory on the node db2. May be it is configured to be at a different location.
Ok, the @2i directory does not exist on db2. And this is the definition of one of the indexes:
CREATE INDEX `def_type_2` ON `data`(`type`) WITH { "defer_build":true, "nodes":[ "db2.dalsgaard-data.dk:8091" ] }
… and that should mean it was created on db2, or…???
Where do you think I should search?
Edit: After having changed my sync gateway to use indexes I can see that half of the indexes were created on db1 and the other half on db2 - but still no change to the directory…
So apparently the server has set another path for the indexes. But, the indexes originally were in the same location - and the directories also exist under /home/couchbase - but haven’t been modified since november 2018?? How could this happen?
Can I remove the indexes (and the directories that remain) and then recreate them? There is no need for old index-files to take up space on the server
What controls where the indexes are created? Something must have changed along the lines…???
During the initial node setup, you have the option of choosing the disk paths for data/index etc. I don’t know exactly what steps were followed.
You can remove the old indexes/directories on disk which the server is not using as per the config. You don’t need to create anything. The server should have created the directories it is currently using.
Removed the node, manually removed the data directories, and readded the node (adjusting the paths) - and everything seems to be on the right path now.
