Where am I wasting my memory for n1ql index?

SQL++
1

Hello,

When I look at the server and check memory usage with “top” command, indexer process is using 60% memory.
We’ve assigned 12GB for “Index RAM Quota”.

We have two index servers and I have the following indexes on one of the servers. This server is comsuming 60% of the system memory by “indexer” process. How is it using 60%? I don’t even have 2 GB indexes in this particular server.

idx_attr_form
    disk size: 12.2 MB
    data size: 217 KB
idx_c_client
    disk size: 1.86 GB
    data size: 842 MB
idx_p_type_client
    disk size: 17.2 MB
    data size: 220 KB
idx_p_type_id
    disk size: 17.2 MB
    data size: 219 KB
idx_p_type_slug
    disk size: 17.2 MB
    data size: 220 KB
2

@moon0326, which couchbase version are you using and what is the storage type(standard or memory optimized)?

3

We are one 4.5.1 and standard memory.

4

Can you run the below 2 curl command on index service node and share the output

curl <host>:9102/stats -u <username:password>
curl <host>:9102/stats/mem -u <username:password>
5

stats

{"catalog:idx_attribute_form:avg_scan_latency":0,"catalog:idx_attribute_form:avg_scan_request_alloc_latency":3105,"catalog:idx_attribute_form:avg_scan_request_init_latency":47323,"catalog:idx_attribute_form:avg_scan_request_latency":347430,"catalog:idx_attribute_form:avg_scan_wait_latency":0,"catalog:idx_attribute_form:avg_ts_interval":662318646,"catalog:idx_attribute_form:avg_ts_items_count":3,"catalog:idx_attribute_form:build_progress":100,"catalog:idx_attribute_form:client_cancel_errcount":0,"catalog:idx_attribute_form:data_size":223070,"catalog:idx_attribute_form:delete_bytes":0,"catalog:idx_attribute_form:disk_load_duration":0,"catalog:idx_attribute_form:disk_size":18120704,"catalog:idx_attribute_form:disk_store_duration":0,"catalog:idx_attribute_form:flush_queue_size":0,"catalog:idx_attribute_form:frag_percent":98,"catalog:idx_attribute_form:get_bytes":4064,"catalog:idx_attribute_form:insert_bytes":5023,"catalog:idx_attribute_form:items_count":53,"catalog:idx_attribute_form:not_ready_errcount":0,"catalog:idx_attribute_form:num_commits":251625,"catalog:idx_attribute_form:num_compactions":15,"catalog:idx_attribute_form:num_completed_requests":1,"catalog:idx_attribute_form:num_docs_indexed":95973326,"catalog:idx_attribute_form:num_docs_pending":0,"catalog:idx_attribute_form:num_docs_processed":98974896,"catalog:idx_attribute_form:num_docs_queued":0,"catalog:idx_attribute_form:num_flush_queued":95973326,"catalog:idx_attribute_form:num_items_flushed":53,"catalog:idx_attribute_form:num_items_restored":0,"catalog:idx_attribute_form:num_last_snapshot_reply":0,"catalog:idx_attribute_form:num_requests":1,"catalog:idx_attribute_form:num_rows_returned":0,"catalog:idx_attribute_form:num_snapshot_waiters":0,"catalog:idx_attribute_form:num_snapshots":251626,"catalog:idx_attribute_form:scan_bytes_read":0,"catalog:idx_attribute_form:scan_wait_duration":0,"catalog:idx_attribute_form:since_last_snapshot":190004437,"catalog:idx_attribute_form:timings/dcp_getseqs":"0 0 0","catalog:idx_attribute_form:timings/scan_pipeline_iterate":"1 246329 60677976241","catalog:idx_attribute_form:timings/storage_clone_handle":"1 19558 382515364","catalog:idx_attribute_form:timings/storage_commit":"251625 214534415529 1866763197129964973","catalog:idx_attribute_form:timings/storage_del":"0 0 0","catalog:idx_attribute_form:timings/storage_get":"95973326 603833299105 1110441352360355217","catalog:idx_attribute_form:timings/storage_info":"1006503 8516206409 215400204780669747","catalog:idx_attribute_form:timings/storage_iterator_next":"53 91151 176998865","catalog:idx_attribute_form:timings/storage_meta_get":"251639 134127816966 6593339143126683498","catalog:idx_attribute_form:timings/storage_meta_set":"251625 277953003591 -7845062935330233367","catalog:idx_attribute_form:timings/storage_new_iterator":"1 37310 1392036100","catalog:idx_attribute_form:timings/storage_persist_snapshot_create":"251625 4774384273 26657221935674715","catalog:idx_attribute_form:timings/storage_set":"106 1724239 47436152809","catalog:idx_attribute_form:timings/storage_snapshot_close":"1 4670 21808900","catalog:idx_attribute_form:timings/storage_snapshot_create":"1 20002 400080004","catalog:idx_attribute_form:total_scan_duration":0,"catalog:idx_attribute_form:total_scan_request_duration":347430,"catalog:idx_creative_clientId:avg_scan_latency":110348705,"catalog:idx_creative_clientId:avg_scan_request_alloc_latency":4442,"catalog:idx_creative_clientId:avg_scan_request_init_latency":221072,"catalog:idx_creative_clientId:avg_scan_request_latency":110577533,"catalog:idx_creative_clientId:avg_scan_wait_latency":4309,"catalog:idx_creative_clientId:avg_ts_interval":660534481,"catalog:idx_creative_clientId:avg_ts_items_count":3,"catalog:idx_creative_clientId:build_progress":100,"catalog:idx_creative_clientId:client_cancel_errcount":394772,"catalog:idx_creative_clientId:data_size":884436311,"catalog:idx_creative_clientId:delete_bytes":0,"catalog:idx_creative_clientId:disk_load_duration":0,"catalog:idx_creative_clientId:disk_size":2599763968,"catalog:idx_creative_clientId:disk_store_duration":0,"catalog:idx_creative_clientId:flush_queue_size":0,"catalog:idx_creative_clientId:frag_percent":65,"catalog:idx_creative_clientId:get_bytes":178601815928,"catalog:idx_creative_clientId:insert_bytes":363381414,"catalog:idx_creative_clientId:items_count":3475121,"catalog:idx_creative_clientId:not_ready_errcount":0,"catalog:idx_creative_clientId:num_commits":251625,"catalog:idx_creative_clientId:num_compactions":15,"catalog:idx_creative_clientId:num_completed_requests":395231,"catalog:idx_creative_clientId:num_docs_indexed":95973326,"catalog:idx_creative_clientId:num_docs_pending":0,"catalog:idx_creative_clientId:num_docs_processed":98974896,"catalog:idx_creative_clientId:num_docs_queued":0,"catalog:idx_creative_clientId:num_flush_queued":95973326,"catalog:idx_creative_clientId:num_items_flushed":3475122,"catalog:idx_creative_clientId:num_items_restored":0,"catalog:idx_creative_clientId:num_last_snapshot_reply":0,"catalog:idx_creative_clientId:num_requests":395231,"catalog:idx_creative_clientId:num_rows_returned":2232907842,"catalog:idx_creative_clientId:num_snapshot_waiters":0,"catalog:idx_creative_clientId:num_snapshots":301632,"catalog:idx_creative_clientId:scan_bytes_read":76701057155,"catalog:idx_creative_clientId:scan_wait_duration":1703138888,"catalog:idx_creative_clientId:since_last_snapshot":190039758,"catalog:idx_creative_clientId:timings/dcp_getseqs":"0 0 0","catalog:idx_creative_clientId:timings/scan_pipeline_iterate":"395231 43502429949905 7743587404600693235","catalog:idx_creative_clientId:timings/storage_clone_handle":"395231 8921771344 171686606979919274","catalog:idx_creative_clientId:timings/storage_commit":"251625 348957938264 7965994663061453178","catalog:idx_creative_clientId:timings/storage_del":"0 0 0","catalog:idx_creative_clientId:timings/storage_get":"95973326 744809001482 2403818550949698072","catalog:idx_creative_clientId:timings/storage_info":"1106515 7758910323 117073256381031603","catalog:idx_creative_clientId:timings/storage_iterator_next":"2232512964 8505555206228 1097497792602423004","catalog:idx_creative_clientId:timings/storage_meta_get":"251639 142270767651 6685593404410197717","catalog:idx_creative_clientId:timings/storage_meta_set":"251625 453828944366 -8466979348101988952","catalog:idx_creative_clientId:timings/storage_new_iterator":"395231 16285698191 125904426641571361","catalog:idx_creative_clientId:timings/storage_persist_snapshot_create":"251625 11261116787 147736445320934209","catalog:idx_creative_clientId:timings/storage_set":"6950244 238826640312 -7417766757087178846","catalog:idx_creative_clientId:timings/storage_snapshot_close":"50007 16707796934 444287657546639266","catalog:idx_creative_clientId:timings/storage_snapshot_create":"50007 1228013608 95810622837102","catalog:idx_creative_clientId:total_scan_duration":43613229256686,"catalog:idx_creative_clientId:total_scan_request_duration":43703668959921,"catalog:idx_product_type_clientId:avg_scan_latency":600160,"catalog:idx_product_type_clientId:avg_scan_request_alloc_latency":5983,"catalog:idx_product_type_clientId:avg_scan_request_init_latency":346450,"catalog:idx_product_type_clientId:avg_scan_request_latency":949444,"catalog:idx_product_type_clientId:avg_scan_wait_latency":3345,"catalog:idx_product_type_clientId:avg_ts_interval":664479342,"catalog:idx_product_type_clientId:avg_ts_items_count":3,"catalog:idx_product_type_clientId:build_progress":100,"catalog:idx_product_type_clientId:client_cancel_errcount":804,"catalog:idx_product_type_clientId:data_size":225601,"catalog:idx_product_type_clientId:delete_bytes":0,"catalog:idx_product_type_clientId:disk_load_duration":0,"catalog:idx_product_type_clientId:disk_size":18124800,"catalog:idx_product_type_clientId:disk_store_duration":0,"catalog:idx_product_type_clientId:flush_queue_size":0,"catalog:idx_product_type_clientId:frag_percent":98,"catalog:idx_product_type_clientId:get_bytes":50913854604,"catalog:idx_product_type_clientId:insert_bytes":4338,"catalog:idx_product_type_clientId:items_count":42,"catalog:idx_product_type_clientId:not_ready_errcount":0,"catalog:idx_product_type_clientId:num_commits":251625,"catalog:idx_product_type_clientId:num_compactions":15,"catalog:idx_product_type_clientId:num_completed_requests":15389265,"catalog:idx_product_type_clientId:num_docs_indexed":95973326,"catalog:idx_product_type_clientId:num_docs_pending":0,"catalog:idx_product_type_clientId:num_docs_processed":98974896,"catalog:idx_product_type_clientId:num_docs_queued":0,"catalog:idx_product_type_clientId:num_flush_queued":95973326,"catalog:idx_product_type_clientId:num_items_flushed":42,"catalog:idx_product_type_clientId:num_items_restored":0,"catalog:idx_product_type_clientId:num_last_snapshot_reply":0,"catalog:idx_product_type_clientId:num_requests":15389265,"catalog:idx_product_type_clientId:num_rows_returned":638142285,"catalog:idx_product_type_clientId:num_snapshot_waiters":0,"catalog:idx_product_type_clientId:num_snapshots":251626,"catalog:idx_product_type_clientId:scan_bytes_read":23248898910,"catalog:idx_product_type_clientId:scan_wait_duration":51486154941,"catalog:idx_product_type_clientId:since_last_snapshot":200047531,"catalog:idx_product_type_clientId:timings/dcp_getseqs":"0 0 0","catalog:idx_product_type_clientId:timings/scan_pipeline_iterate":"15389265 4533628923839 5922501881310363057","catalog:idx_product_type_clientId:timings/storage_clone_handle":"15389265 490920496399 4207335156886680363","catalog:idx_product_type_clientId:timings/storage_commit":"251625 219714610197 6194360367248486851","catalog:idx_product_type_clientId:timings/storage_del":"0 0 0","catalog:idx_product_type_clientId:timings/storage_get":"95973326 610367257237 -7359296775475645773","catalog:idx_product_type_clientId:timings/storage_info":"1006503 8333775633 93262021582039265","catalog:idx_product_type_clientId:timings/storage_iterator_next":"638142285 1465049712217 3203715597001005109","catalog:idx_product_type_clientId:timings/storage_meta_get":"251638 132398748903 6148885918467651697","catalog:idx_product_type_clientId:timings/storage_meta_set":"251625 262478077101 -1952757056319378429","catalog:idx_product_type_clientId:timings/storage_new_iterator":"15389265 592556555308 -6264895180805866260","catalog:idx_product_type_clientId:timings/storage_persist_snapshot_create":"251625 4932149384 27424182182090242","catalog:idx_product_type_clientId:timings/storage_set":"84 11155617 80400800034055","catalog:idx_product_type_clientId:timings/storage_snapshot_close":"1 5331 28419561","catalog:idx_product_type_clientId:timings/storage_snapshot_create":"1 29765 885955225","catalog:idx_product_type_clientId:total_scan_duration":9236036167950,"catalog:idx_product_type_clientId:total_scan_request_duration":14611249360519,"catalog:idx_product_type_id:avg_scan_latency":0,"catalog:idx_product_type_id:avg_scan_request_alloc_latency":0,"catalog:idx_product_type_id:avg_scan_request_init_latency":0,"catalog:idx_product_type_id:avg_scan_request_latency":0,"catalog:idx_product_type_id:avg_scan_wait_latency":0,"catalog:idx_product_type_id:avg_ts_interval":664314845,"catalog:idx_product_type_id:avg_ts_items_count":3,"catalog:idx_product_type_id:build_progress":100,"catalog:idx_product_type_id:client_cancel_errcount":0,"catalog:idx_product_type_id:data_size":225371,"catalog:idx_product_type_id:delete_bytes":0,"catalog:idx_product_type_id:disk_load_duration":0,"catalog:idx_product_type_id:disk_size":18124800,"catalog:idx_product_type_id:disk_store_duration":0,"catalog:idx_product_type_id:flush_queue_size":0,"catalog:idx_product_type_id:frag_percent":98,"catalog:idx_product_type_id:get_bytes":936,"catalog:idx_product_type_id:insert_bytes":4108,"catalog:idx_product_type_id:items_count":42,"catalog:idx_product_type_id:not_ready_errcount":0,"catalog:idx_product_type_id:num_commits":251625,"catalog:idx_product_type_id:num_compactions":15,"catalog:idx_product_type_id:num_completed_requests":0,"catalog:idx_product_type_id:num_docs_indexed":95973326,"catalog:idx_product_type_id:num_docs_pending":0,"catalog:idx_product_type_id:num_docs_processed":98974896,"catalog:idx_product_type_id:num_docs_queued":0,"catalog:idx_product_type_id:num_flush_queued":95973326,"catalog:idx_product_type_id:num_items_flushed":42,"catalog:idx_product_type_id:num_items_restored":0,"catalog:idx_product_type_id:num_last_snapshot_reply":0,"catalog:idx_product_type_id:num_requests":0,"catalog:idx_product_type_id:num_rows_returned":0,"catalog:idx_product_type_id:num_snapshot_waiters":0,"catalog:idx_product_type_id:num_snapshots":251626,"catalog:idx_product_type_id:scan_bytes_read":0,"catalog:idx_product_type_id:scan_wait_duration":0,"catalog:idx_product_type_id:since_last_snapshot":200074733,"catalog:idx_product_type_id:timings/dcp_getseqs":"0 0 0","catalog:idx_product_type_id:timings/scan_pipeline_iterate":"0 0 0","catalog:idx_product_type_id:timings/storage_clone_handle":"0 0 0","catalog:idx_product_type_id:timings/storage_commit":"251625 167188610082 5636355575449386182","catalog:idx_product_type_id:timings/storage_del":"0 0 0","catalog:idx_product_type_id:timings/storage_get":"95973326 606363886574 240786279114454412","catalog:idx_product_type_id:timings/storage_info":"1006503 8029738789 63674108510008599","catalog:idx_product_type_id:timings/storage_iterator_next":"0 0 0","catalog:idx_product_type_id:timings/storage_meta_get":"251639 132685743518 5942129720658775610","catalog:idx_product_type_id:timings/storage_meta_set":"251625 265186813701 -9070388326706379383","catalog:idx_product_type_id:timings/storage_new_iterator":"0 0 0","catalog:idx_product_type_id:timings/storage_persist_snapshot_create":"251625 4744471031 26890121458942745","catalog:idx_product_type_id:timings/storage_set":"84 24924667 567458374470957","catalog:idx_product_type_id:timings/storage_snapshot_close":"1 6732 45319824","catalog:idx_product_type_id:timings/storage_snapshot_create":"1 16732 279959824","catalog:idx_product_type_id:total_scan_duration":0,"catalog:idx_product_type_id:total_scan_request_duration":0,"catalog:idx_product_type_slug:avg_scan_latency":409648,"catalog:idx_product_type_slug:avg_scan_request_alloc_latency":5950,"catalog:idx_product_type_slug:avg_scan_request_init_latency":369418,"catalog:idx_product_type_slug:avg_scan_request_latency":780690,"catalog:idx_product_type_slug:avg_scan_wait_latency":3255,"catalog:idx_product_type_slug:avg_ts_interval":664862090,"catalog:idx_product_type_slug:avg_ts_items_count":3,"catalog:idx_product_type_slug:build_progress":100,"catalog:idx_product_type_slug:client_cancel_errcount":276,"catalog:idx_product_type_slug:data_size":225737,"catalog:idx_product_type_slug:delete_bytes":0,"catalog:idx_product_type_slug:disk_load_duration":0,"catalog:idx_product_type_slug:disk_size":18124800,"catalog:idx_product_type_slug:disk_store_duration":0,"catalog:idx_product_type_slug:flush_queue_size":0,"catalog:idx_product_type_slug:frag_percent":98,"catalog:idx_product_type_slug:get_bytes":1970194413,"catalog:idx_product_type_slug:insert_bytes":4474,"catalog:idx_product_type_slug:items_count":42,"catalog:idx_product_type_slug:not_ready_errcount":0,"catalog:idx_product_type_slug:num_commits":251625,"catalog:idx_product_type_slug:num_compactions":15,"catalog:idx_product_type_slug:num_completed_requests":12168665,"catalog:idx_product_type_slug:num_docs_indexed":95973326,"catalog:idx_product_type_slug:num_docs_pending":0,"catalog:idx_product_type_slug:num_docs_processed":98974896,"catalog:idx_product_type_slug:num_docs_queued":0,"catalog:idx_product_type_slug:num_flush_queued":95973326,"catalog:idx_product_type_slug:num_items_flushed":42,"catalog:idx_product_type_slug:num_items_restored":0,"catalog:idx_product_type_slug:num_last_snapshot_reply":0,"catalog:idx_product_type_slug:num_requests":12168665,"catalog:idx_product_type_slug:num_rows_returned":12168657,"catalog:idx_product_type_slug:num_snapshot_waiters":0,"catalog:idx_product_type_slug:num_snapshots":251626,"catalog:idx_product_type_slug:scan_bytes_read":458742768,"catalog:idx_product_type_slug:scan_wait_duration":39613206210,"catalog:idx_product_type_slug:since_last_snapshot":200108650,"catalog:idx_product_type_slug:timings/dcp_getseqs":"0 0 0","catalog:idx_product_type_slug:timings/scan_pipeline_iterate":"12168665 1595559371306 9042953181167081442","catalog:idx_product_type_slug:timings/storage_clone_handle":"12168665 423317790032 2746147171866499396","catalog:idx_product_type_slug:timings/storage_commit":"251625 172569696007 7621471478104140137","catalog:idx_product_type_slug:timings/storage_del":"0 0 0","catalog:idx_product_type_slug:timings/storage_get":"95973326 593721214388 3140939804870123082","catalog:idx_product_type_slug:timings/storage_info":"1006503 8503048717 107661181739338887","catalog:idx_product_type_slug:timings/storage_iterator_next":"12168657 79683231185 4025098269226674187","catalog:idx_product_type_slug:timings/storage_meta_get":"251638 138312986651 6886970325270101193","catalog:idx_product_type_slug:timings/storage_meta_set":"251625 270158240713 -2782801717985041025","catalog:idx_product_type_slug:timings/storage_new_iterator":"12168665 467349103396 6956281296486032806","catalog:idx_product_type_slug:timings/storage_persist_snapshot_create":"251625 4752853476 32463823135030874","catalog:idx_product_type_slug:timings/storage_set":"84 5016057 15531796240539","catalog:idx_product_type_slug:timings/storage_snapshot_close":"1 3123 9753129","catalog:idx_product_type_slug:timings/storage_snapshot_create":"1 14254 203176516","catalog:idx_product_type_slug:total_scan_duration":4984877016432,"catalog:idx_product_type_slug:total_scan_request_duration":9499967151464,"catalog:mutation_queue_size":0,"catalog:num_mutations_queued":95973326,"catalog:num_nonalign_ts":4,"catalog:timings/dcp_getseqs":"267557 444662891660 -6580369942802371436","catalog:ts_queue_size":0,"index_not_found_errcount":0,"indexer_state":"Active","memory_quota":12884901888,"memory_used":3627227136,"memory_used_queue":0,"memory_used_storage":2649767936,"needs_restart":false,"num_connections":88,"storage_mode":"forestdb","timings/stats_response":"1903417 2044671578229 2969677074643515393","uptime":"380h0m52.012603425s"}

mem

{"Alloc":387439824,"TotalAlloc":11152402353432,"Sys":10771427976,"Lookups":5854283,"Mallocs":143881047378,"Frees":143868592728,"HeapAlloc":387439824,"HeapSys":10263887872,"HeapIdle":9532211200,"HeapInuse":731676672,"HeapReleased":9414795264,"HeapObjects":12454650,"StackInuse":4751360,"StackSys":4751360,"MSpanInuse":15630000,"MSpanSys":149520384,"MCacheInuse":9600,"MCacheSys":16384,"BuckHashSys":2621703,"GCSys":322508800,"OtherSys":28121473,"NextGC":507455442,"LastGC":1477601027846438984,"PauseTotalNs":152497793657,"PauseNs":[989673,782787,799843,998155,851400,921849,796150,827202,803227,1031741,1003011,839871,917549,866620,1051629,919136,1009881,995186,588669,996745,837332,1182973,826269,815476,811562,774807,860009,841224,796664,816088,830491,816433,794342,953362,878072,819053,944469,813822,780470,763925,618809,1047366,824861,1002187,986592,800022,1010575,1069058,1197248,841156,835120,911450,849599,1064204,894591,792069,1022389,919081,813042,1161784,861153,997869,861692,886945,1004723,826513,784828,1050890,767720,787314,824644,836417,918498,884036,983498,796039,907525,810383,831987,804863,837629,1091337,1070206,794711,833813,1090743,922005,944658,945329,1041491,936329,803488,799752,960584,822110,822748,806544,832330,816190,1114926,1004430,879215,1175636,968940,1057762,802687,843354,1069332,906821,996840,3288889,801115,1073543,849881,1057984,819326,782636,808058,790727,1073592,846714,1188791,749735,844942,963406,811126,815238,953810,1070369,1332407,1013542,809131,792577,813027,956595,836467,899566,863661,825518,1033434,1000458,958695,822435,862923,814823,1008143,1227041,812202,776209,1157365,861176,862551,1001935,772878,817492,1000460,1037111,1016124,592101,875652,823700,801214,822913,1047762,803024,975976,817400,772450,801892,999400,1091704,1106124,1007667,987842,883298,1006365,841329,1030397,977894,987198,782543,762503,821005,785873,882461,973919,1319400,1103300,773559,1015731,774460,976440,848784,892724,974997,888782,923836,881504,886540,993815,854151,930900,801099,1000477,864147,820874,828566,972186,855762,990754,968360,821274,817429,990245,885981,977975,805723,618523,738142,847054,1244902,837667,950450,846788,811615,977673,800365,861795,830808,1031850,987555,1022514,825122,976050,803770,1003285,810438,983133,778627,821158,1182461,996719,1219714,1024789,1015538,769926,1029229,800695,1041010,1144091,868595,1054432,977479,833777,889619,968380],"PauseEnd":[1477594375106586863,1477594420439293144,1477594465568995953,1477594509264461154,1477594550468903978,1477594595089209742,1477594635417381074,1477594675194388147,1477594720877699937,1477594758383974561,1477594799545669279,1477594844677412626,1477594891266260351,1477594936388674173,1477594974871753783,1477595012291072993,1477595057538344396,1477595099704445570,1477595144301191881,1477595175502147934,1477595214013875592,1477595253427807910,1477595297670770454,1477595339063893287,1477595378065030148,1477595420708133297,1477595464972005696,1477595508665423793,1477595544660197851,1477595586063132186,1477595632266367115,1477595675091945389,1477595718091574574,1477595760215824982,1477595803867463833,1477595841739884740,1477595882057477517,1477595926068527353,1477595967064215861,1477596010240264334,1477596050337989174,1477596094703453730,1477596136862071618,1477596181744364438,1477596227143602285,1477596270996790677,1477596315862049312,1477596361040535781,1477596404871844265,1477596445452393954,1477596487068011324,1477596533670957522,1477596569245432794,1477596612744201931,1477596656263310609,1477596702263620852,1477596743866773774,1477596781652561885,1477596810263991903,1477596852349636107,1477596892681992937,1477596933845025231,1477596967744653517,1477597008487843646,1477597050741128386,1477597092460405978,1477597133671497219,1477597168663076386,1477597210864894028,1477597251228867982,1477597292541622699,1477597330848614128,1477597373736425534,1477597414689119643,1477597455661110808,1477597499582549450,1477597540960551992,1477597582192799504,1477597627472404024,1477597674267639607,1477597715468248498,1477597760822309282,1477597803347543410,1477597843991140952,1477597885235814875,1477597930381689911,1477597973198281061,1477598017301257741,1477598063152112776,1477598105860709767,1477598148447729683,1477598193863381514,1477598232805015403,1477598275247902684,1477598317259545044,1477598357575055543,1477598399873234760,1477598441060884387,1477598484472928201,1477598528278181235,1477598571224861737,1477598615355750619,1477598659862402741,1477598704769863693,1477598750647607198,1477598795865104004,1477598842058949025,1477598887846698268,1477598935068064270,1477598976544653009,1477599022468126286,1477599060569850028,1477599106422191404,1477599155057844945,1477599200431439691,1477599245094136705,1477599287118786363,1477599329161327713,1477599374576871647,1477599420672556042,1477599465060880016,1477599508220151240,1477599554871802229,1477599597668936769,1477599638065317825,1477599679908576319,1477599725091197542,1477599769154049751,1477599811150173237,1477599852857210312,1477599898103437101,1477599942575288515,1477599986459595728,1477600032873165001,1477600077700873272,1477600122782556880,1477600168055343118,1477600212859208509,1477600254490996886,1477600295862030426,1477600341821247373,1477600386504409196,1477600430063237549,1477600472247639539,1477600517674850090,1477600563183871216,1477600607464048961,1477600652099633257,1477600710068301259,1477600756908434961,1477600802350153354,1477600846520030135,1477600890618270437,1477600935136784333,1477600983526169384,1477601027846438984,1477590223586428300,1477590264699901230,1477590305464307996,1477590345453753391,1477590387336978271,1477590428665493428,1477590470451100975,1477590511068855488,1477590552661670911,1477590596292373641,1477590635260457498,1477590679808909818,1477590713793112169,1477590754783102663,1477590797939179769,1477590841622990908,1477590886102744678,1477590931847767210,1477590973645801062,1477591012246690171,1477591056868437787,1477591101744831230,1477591145045829155,1477591191501940392,1477591231599640672,1477591264289449921,1477591308181261209,1477591353570995627,1477591395671021675,1477591433861601922,1477591471342780907,1477591510499869502,1477591552044294724,1477591581742693740,1477591619478152357,1477591660661493623,1477591703449354383,1477591739455927078,1477591779825866731,1477591816199188872,1477591858266601715,1477591898463000791,1477591939866744567,1477591982510357656,1477592024863659769,1477592067886534545,1477592109871668006,1477592152630574554,1477592192458199509,1477592234058971840,1477592277052677425,1477592322261904381,1477592365650761851,1477592404467690934,1477592445488840935,1477592487830681495,1477592533271070240,1477592574660190706,1477592617944643230,1477592660462155638,1477592703473853852,1477592744263683651,1477592786047003148,1477592829625025977,1477592872893901251,1477592915090078987,1477592957253220692,1477592995045204522,1477593025386848317,1477593061905545420,1477593104570720883,1477593146614192911,1477593185668997799,1477593227044005305,1477593269712440263,1477593311865283629,1477593351266253099,1477593390853576640,1477593433462077418,1477593476222826168,1477593520261245941,1477593556461497158,1477593593865595163,1477593635049697057,1477593679846699321,1477593722230619970,1477593766111860689,1477593811140831711,1477593849508958187,1477593895088849512,1477593941267597914,1477593989269184706,1477594033143743643,1477594076243430286,1477594114727438112,1477594155450470364,1477594200413874715,1477594241933732314,1477594287464491367,1477594331537737325],"NumGC":40604,"GCCPUFraction":0.004473375383965203,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":3457271073,"Frees":3457246505},{"Size":16,"Mallocs":42409561533,"Frees":42397615320},{"Size":32,"Mallocs":31477138681,"Frees":31476803450},{"Size":48,"Mallocs":7064911513,"Frees":7064864198},{"Size":64,"Mallocs":4670411468,"Frees":4670382448},{"Size":80,"Mallocs":4551510930,"Frees":4551486473},{"Size":96,"Mallocs":1662530620,"Frees":1662524753},{"Size":112,"Mallocs":316450002,"Frees":316446892},{"Size":128,"Mallocs":76484220,"Frees":76483418},{"Size":144,"Mallocs":78824840,"Frees":78823876},{"Size":160,"Mallocs":47586407,"Frees":47586034},{"Size":176,"Mallocs":1458643461,"Frees":1458630928},{"Size":192,"Mallocs":316292936,"Frees":316292442},{"Size":208,"Mallocs":294613084,"Frees":294609600},{"Size":224,"Mallocs":57268734,"Frees":57267652},{"Size":240,"Mallocs":59235094,"Frees":59234813},{"Size":256,"Mallocs":47998495,"Frees":47998146},{"Size":288,"Mallocs":144559259,"Frees":144557428},{"Size":320,"Mallocs":16136007,"Frees":16135719},{"Size":352,"Mallocs":71825824,"Frees":71824953},{"Size":384,"Mallocs":8218090,"Frees":8216266},{"Size":416,"Mallocs":36405041,"Frees":36404375},{"Size":448,"Mallocs":8962640,"Frees":8962478},{"Size":480,"Mallocs":5795722,"Frees":5795644},{"Size":512,"Mallocs":82860828,"Frees":82859778},{"Size":576,"Mallocs":5592386,"Frees":5592253},{"Size":640,"Mallocs":511501,"Frees":511458},{"Size":704,"Mallocs":19446862,"Frees":19446567},{"Size":768,"Mallocs":14193834,"Frees":14193579},{"Size":896,"Mallocs":32483911,"Frees":32483177},{"Size":1024,"Mallocs":43129549,"Frees":43129041},{"Size":1152,"Mallocs":17962836,"Frees":17962511},{"Size":1280,"Mallocs":1249136,"Frees":1249107},{"Size":1408,"Mallocs":1215463,"Frees":1215431},{"Size":1536,"Mallocs":26392733,"Frees":26392285},{"Size":1664,"Mallocs":540565,"Frees":540530},{"Size":2048,"Mallocs":30613268,"Frees":30612874},{"Size":2304,"Mallocs":17226696,"Frees":17226407},{"Size":2560,"Mallocs":14634392,"Frees":14634122},{"Size":2816,"Mallocs":685721,"Frees":685707},{"Size":3072,"Mallocs":372225,"Frees":372210},{"Size":3328,"Mallocs":286076,"Frees":286050},{"Size":4096,"Mallocs":38243430,"Frees":38242773},{"Size":4608,"Mallocs":3958186,"Frees":3958111},{"Size":5376,"Mallocs":13414631,"Frees":13414412},{"Size":6144,"Mallocs":13921689,"Frees":13921435},{"Size":6400,"Mallocs":221,"Frees":221},{"Size":6656,"Mallocs":217,"Frees":216},{"Size":6912,"Mallocs":226,"Frees":226},{"Size":8192,"Mallocs":333270977,"Frees":333267150},{"Size":8448,"Mallocs":486321,"Frees":486321},{"Size":8704,"Mallocs":16604712,"Frees":16604417},{"Size":9472,"Mallocs":1169828,"Frees":1169800},{"Size":10496,"Mallocs":1214509,"Frees":1214509},{"Size":12288,"Mallocs":12945143,"Frees":12944940},{"Size":13568,"Mallocs":594221,"Frees":594206},{"Size":14080,"Mallocs":16,"Frees":16},{"Size":16384,"Mallocs":136030527,"Frees":136028967},{"Size":16640,"Mallocs":1125,"Frees":1125},{"Size":17664,"Mallocs":15442584,"Frees":15442342}]}

Thanks!

6

Indexer is reporting memory usage of 3.5GB(2.5 in storage and rest in overheads/temporary allocations).

I see 9.5GB of heap released back to OS by golang. golang is not aggressive in releasing memory back to OS. That may have lead to higher peak usage. Do you still see 60% usage reported by top?

7

Thank you @deepkaran.salooja

Yeah, it is still reporting 60.3%. It’s been at least a couple of days as far as I know.

Thanks,
Moon

8

With standard index, the storage buffer cache is allocated in full at the start of the process. @deepkaran.salooja does this not explain the memory usage? If you would like memory to grow in line with usage, and your use case fits all-in-memory constraint, you could use Memory Optimized Index mode.

9

@siri, full storage buffer cache allocation at the start was in 4.0.x. With 4.5.x, the buffer cache is allocated on demand. The storage buffer cache is being reported as 2.5GB here. We have a pending improvement MB-19895 to report all memory usage statistics from storage. I guess storage memory usage outside of buffer cache is the cause here as everything else is already accounted for.

I agree that Memory Optimized mode is preferred for predictable memory usage.

10

Thank you both @siri @deepkaran.salooja.

@deepkaran.salooja Is there anything I can do to free the memory at this point? The server is still at 60% and it’s not going up or down. I’m not sure if it is normal if it stays for over 4+ days without a reason.

Do you think this is related to OOM issue with indexer?

We had a few of these…

[user:info,2016-10-12T19:24:58.702Z,ns_1@xxx:<0.964.0>:ns_log:crash_consumption_loop:70]Service 'indexer' exited with status 1. Restarting. Messages: fatal error: runtime: out of memory

And we still don’t know why it was killed (why it was out of memory when there were plenty)

11

@moon0326, you can try to kill the indexer process on the node(it will restart on its own) and see if the memory gets released. That’s not the solution but it will help us get an idea if it is being leaked. Please run the above two curl commands before and after killing(once the process is up and all indexes are back online). Also capture the corresponding ‘top’ output for the indexer process.

12

Hi @deepkaran.salooja

Thank you for the suggestions.

Unfortunately, we updated our server to use memory optimized index a few days. It works much better than standard.

It looks like the issues we had before (out of memory) are related to this issue.

13

@moon0326, good to hear that you switched to memory optimized indexes. Do let us know if you face any issues with it.