GSI Indexes growing indefinitely

Couchbase Server
1

Hello,

We’re trying out Couchbase Enterprise Edition 7.6.4 build 5146 but we have an issue that we cannot figure out. The indexes keeps growing until they occupy the whole disk.

The disk is ext4, main bucket contains around 300K documents.

[root@server-instance-v2 @2i]# du -cksh * | sort -rh
138G    total
2.9G    hrs_adv_companyId_meta_id_bookingResult_data_flightOffers0_chargeAmount_5957735829133141493_0.index
2.9G    hrs_adv_companyId_meta_id_8850357005463079428_0.index
2.9G    hrs_adv_combined_bookingRate_userData_exceedsPolicyLimit_companyId_meta_id_policy_id_count_10758791702350556098_0.index
2.9G    hrs_adv_bookingResult_data_flightOffers0_travelerPricings0_userData_1506729552_15380352469523833507_0.index
2.8G    hrs_idx_push_tokens_consolidated_4325257141410386655_0.index
2.8G    hrs_airBookingsConsolidatedIndex_6225744916456278318_0.index
2.6G    shards
2.5G    hrs_idx_hrs_type_11873868772694642082_0.index
2.5G    hrs_idx_emailCode_2_10040514299481959355_0.index
2.5G    hrs_idx_emailCode_1_11150257696180397076_0.index
2.5G    hrs_adv_referenceNumber_class_8653831852766080384_0.index
2.5G    hrs_adv_lower_bookingResult_data_associatedRecords_0_reference_lower238877905_2762152148865054283_0.index
2.4G    hrs_users_stripe_customer_id_12625997587250320491_0.index
2.4G    hrs_userDiscounts_16910555900617828125_0.index
2.4G    hrs_netMarginDiscount_15427557061242532879_0.index
2.4G    hrs_idx_userCode_1628518528997876783_0.index
2.4G    hrs_idx_userCode_1_16627839491205714634_0.index
2.4G    hrs_idx_teams_paymentMethodsIds_2_10578410297871822345_0.index
2.4G    hrs_idx_hrs_usersType_16309675261269761163_0.index
2.4G    hrs_idx_hrs_leadUsers_lower_email_10266024752475512595_0.index
2.4G    hrs_idx_hotel_reviews_1375712546913230184_0.index
2.4G    hrs_idx_companyBookings_dashboard_1765721268087681507_0.index
2.4G    hrs_idx_cheque_payouts_2883791984655483029_0.index
2.4G    hrs_adv_user_company_lower_email_class_7208430063741474232_0.index
2.4G    hrs_adv_status_requesterUserId_teamId_class_16166525188914122845_0.index
2.4G    hrs_adv_status_approverUserId_teamId_teamId_class_11724080686382897949_0.index
2.4G    hrs_adv_status_approverUserId_teamId_class_6603431194532156443_0.index
2.4G    hrs_adv_rewardReceived_checkoutDate_rewardAmount_bookingStatus_class_14572776169283098619_0.index
2.4G    hrs_adv_policyId_class_4450221895644977100_0.index
2.4G    hrs_adv_membership_companyMembership_companyId_membership_companyMem4172941942_408480753678787040_0.index
2.4G    hrs_adv_membership_companyMembership_companyId_lower_lastName_lower_firstName_lower_email_class_14497299893194371315_0.index
2.4G    hrs_adv_membership_companyMembership_companyId_class_membership_companyMembership_inactive_5024682432900706953_0.index
2.4G    hrs_adv_lower_email_class_745936991935331327_0.index
2.4G    hrs_adv_hrsReferenceNumber_class_7517329847179874753_0.index
2.4G    hrs_adv_DISTINCT_approvers_class_10743874032715545476_0.index
2.4G    hrs_adv_display_class_4153957644746654263_0.index
2.4G    hrs_adv_companyId_substr0_creationDate010_class_4243627340668778395_0.index
2.4G    hrs_adv_companyId_class_5648528200537993434_0.index
2.4G    hrs_adv_companyDbId_class_1584265190778598230_0.index
2.4G    hrs_adv_class_substr0_creationDate_010_1_6004944597352805415_0.index
2.4G    hrs_adv_class_companyId2_2171755314841477836_0.index
2.4G    hrs_adv_bookingStatus_userId_tripStartDateTime_class_17998866608919609012_0.index
2.4G    hrs_adv_bookingStatus_checkoutDate_userId_travelerId_class_7112224790140134332_0.index
2.4G    hrs_adv_bookingStatus_checkoutDate_class_10393906880420592856_0.index
2.2G    hrs_adv_inactive_substr0_creationDate010_lower_name_lower_domain_class_7411110214209367795_0.index
1.8G    hrs_#primary_7786860651755249870_0.index
1.7G    hrs_adv_bookingStatus_travelerId_userId_class_4719570424774471264_0.index
1.5G    hrs_adv_lower_companyId_chargeAmount_class_6644134054308316684_0.index
1.1G    hrs_static_content_idx_triphop_mappings_2_751117558843196495_0.index
1001M   hrs_static_content_idx_hrs_static_content_type_4308206713101905357_0.index
955M    hrs_static_content_idx_triphop_properties_2_14104791410250913923_0.index
933M    hrs_static_content_idx_triphop_properties_name_13823893235523116983_0.index
909M    hrs_static_content_hotel_ids_name_country_city_state_lower_13024777128863729841_0.index

If I drop an index and create it again, it drops down to KBytes, and over time it will grow indefinitely again.

image
image2147×548 70 KB

If someone could assist it would be much appreciated.

Thank you,

2

Hi @lenix - did this work on a different version?

3

@mreiche I think it didnt happen on the older version we had before updating to 7.6.4

What other information is needed to diagnose this?

Much appreciated.

4

Still seeking assistance regarding this matter.

See screenshots below, usage on disk does not match with web ui, and it keeps growing.

Also forgot to mention it’s a single node cluster

indexes-on-disk
indexes-on-disk1435×1219 148 KB

indexe-on-webui
indexe-on-webui2151×965 92.7 KB

5

Since you have Enterprise Edition, you can open a ticket with customer support

6

Another observation is that the problem occurs only on a specific bucket thats configured as the root bucket in spring boot. Also what consumes most data is the ‘recovery’ folders inside indexes folders, for example

[user-dev@qa-server-instance-v2 ~]$ sudo ls -lhR /home/user-dev/binaries/couchbase/@2i/hrs_netMarginDiscount_15427557061242532879_0.index
/home/user-dev/binaries/couchbase/@2i/hrs_netMarginDiscount_15427557061242532879_0.index:
total 8.0K
drwxr-x---. 3 couchbase couchbase 4.0K Mar  9 11:28 docIndex
drwxr-x---. 3 couchbase couchbase 4.0K Mar  9 11:28 mainIndex

/home/user-dev/binaries/couchbase/@2i/hrs_netMarginDiscount_15427557061242532879_0.index/docIndex:
total 35M
-rwxr-x---. 1 couchbase couchbase   57 Feb  2 14:44 config.json
-rwxr-x---. 1 couchbase couchbase 8.0K Apr 21 08:48 header.data
-rwxr-x---. 1 couchbase couchbase 1.8G Apr 21 08:48 log.00000000000000.data
drwxr-x---. 2 couchbase couchbase 4.0K Mar  9 11:28 recovery

/home/user-dev/binaries/couchbase/@2i/hrs_netMarginDiscount_15427557061242532879_0.index/docIndex/recovery:
total 1.8G
-rwxr-x---. 1 couchbase couchbase 8.0K Apr 21 08:48 header.data
-rwxr-x---. 1 couchbase couchbase 1.8G Apr 21 08:48 log.00000000000000.data

/home/user-dev/binaries/couchbase/@2i/hrs_netMarginDiscount_15427557061242532879_0.index/mainIndex:
total 35M
-rwxr-x---. 1 couchbase couchbase   57 Feb  2 14:44 config.json
-rwxr-x---. 1 couchbase couchbase 8.0K Apr 21 08:48 header.data
-rwxr-x---. 1 couchbase couchbase 1.8G Apr 21 08:48 log.00000000000000.data
drwxr-x---. 2 couchbase couchbase 4.0K Mar  9 11:28 recovery

/home/user-dev/binaries/couchbase/@2i/hrs_netMarginDiscount_15427557061242532879_0.index/mainIndex/recovery:
total 1.8G
-rwxr-x---. 1 couchbase couchbase 8.0K Apr 21 08:48 header.data
-rwxr-x---. 1 couchbase couchbase 1.8G Apr 21 08:48 log.00000000000000.data

If you see most data is being eaten by the recovery files inside mainIndex and docIndex

I cannot see any errors in indexer.log

But in indexer.log the frag stays 99 for these indexes

2025-04-21T05:08:53.096+00:00 [Info] LSS /home/user-dev/binaries/couchbase/@2i/hrs_adv_companyDbId_class_1584265190778598230_0.index/mainIndex(shard9706047719756208648) : logCleaner: starting... frag 99, data: 78437, used: 16874897 log:(1893270108 - 1910509568) head:1893270108
2025-04-21T05:08:53.096+00:00 [Info] LSS /home/user-dev/binaries/couchbase/@2i/hrs_adv_companyDbId_class_1584265190778598230_0.index/mainIndex(shard9706047719756208648) : logCleaner: completed... frag 99, data: 78437, used: 12258705, disk: 17313188, relocated: 0, rewrites: 1, retries: 0, skipped: 0, cleaned:4329472, log:(1897599580 - 1910583296), disk:(1893270108-1910583296), activeFragRatio:30, activeMaxFragRatio:80, run:1 duration:9 ms elapsed: 9 ms paused:false

All couchbase data is on ext4 partition
/dev/sda ext4 197G 175G 22G 89% /home/user-dev/binaries

I know that XFS is recommended but ext4 should also work right?

Any ideas?