Getting generic network failure randomly while opening buckets using Node.js SDK

Node.js SDK
1

I am randomly seeing a “Generic network failure” error while opening buckets (which I do once at the beginning after starting the server) using Node.js SDK. On restarting the Node.js server a few times, I saw that sometimes the bucket opens successfully and all the operations after that run properly. But, many a times, the requests for opening the buckets fail.
I verified that the ports mentioned at Couchbase Server Ports | Couchbase Docs are open for communication.
Can you help me in finding out the cause of this network issue that happens randomly and guide me to find out what mistake am I making here?
I am adding the logs of NodeJS server after setting LCB_LOGLEVEL=5.

0ms [I065aa9ae] {26} [INFO] (instance - L:470) Effective connection string: http://0.0.0.0/organization?enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.12%20(node%2F10.22.0%3B%20v8%2F6.8.275.32-node.58%3B%20ssl%2F1.1.1g). Bucket=organization
0ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
0ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
0ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.12 (node/10.22.0; v8/6.8.275.32-node.58; ssl/1.1.1g)
106ms [I065aa9ae] {26} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
106ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:8091 to initial HTTP bootstrap list
106ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:11210 to initial CCCP bootstrap list
106ms [I065aa9ae] {26} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
107ms [I065aa9ae] {26} [INFO] (instance - L:469) Version=2.10.6-njs, Changeset=61e274aae48b6edeecb081afec843dbe31d72790
107ms [I065aa9ae] {26} [INFO] (instance - L:470) Effective connection string: http://0.0.0.0/user-config?enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.12%20(node%2F10.22.0%3B%20v8%2F6.8.275.32-node.58%3B%20ssl%2F1.1.1g). Bucket=user-config
107ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
107ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
107ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.12 (node/10.22.0; v8/6.8.275.32-node.58; ssl/1.1.1g)
114ms [I065aa9ae] {26} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
114ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:8091 to initial HTTP bootstrap list
114ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:11210 to initial CCCP bootstrap list
114ms [I065aa9ae] {26} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
114ms [I065aa9ae] {26} [INFO] (instance - L:469) Version=2.10.6-njs, Changeset=61e274aae48b6edeecb081afec843dbe31d72790
114ms [I065aa9ae] {26} [INFO] (instance - L:470) Effective connection string: http://0.0.0.0/user-projects?enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.12%20(node%2F10.22.0%3B%20v8%2F6.8.275.32-node.58%3B%20ssl%2F1.1.1g). Bucket=user-projects
114ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
114ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
114ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.12 (node/10.22.0; v8/6.8.275.32-node.58; ssl/1.1.1g)
122ms [I065aa9ae] {26} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
122ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:8091 to initial HTTP bootstrap list
122ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:11210 to initial CCCP bootstrap list
122ms [I065aa9ae] {26} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
122ms [I065aa9ae] {26} [INFO] (instance - L:469) Version=2.10.6-njs, Changeset=61e274aae48b6edeecb081afec843dbe31d72790
122ms [I065aa9ae] {26} [INFO] (instance - L:470) Effective connection string: http://0.0.0.0/events?enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.12%20(node%2F10.22.0%3B%20v8%2F6.8.275.32-node.58%3B%20ssl%2F1.1.1g). Bucket=events
122ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
122ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
122ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.12 (node/10.22.0; v8/6.8.275.32-node.58; ssl/1.1.1g)
125ms [I065aa9ae] {26} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
125ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:8091 to initial HTTP bootstrap list
125ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:11210 to initial CCCP bootstrap list
125ms [I065aa9ae] {26} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
126ms [I065aa9ae] {26} [INFO] (instance - L:469) Version=2.10.6-njs, Changeset=61e274aae48b6edeecb081afec843dbe31d72790
126ms [I065aa9ae] {26} [INFO] (instance - L:470) Effective connection string: http://0.0.0.0/notifications?enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.12%20(node%2F10.22.0%3B%20v8%2F6.8.275.32-node.58%3B%20ssl%2F1.1.1g). Bucket=notifications
126ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
126ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
126ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.12 (node/10.22.0; v8/6.8.275.32-node.58; ssl/1.1.1g)
129ms [I065aa9ae] {26} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
129ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:8091 to initial HTTP bootstrap list
129ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:11210 to initial CCCP bootstrap list
129ms [I065aa9ae] {26} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
e[32m[2022-05-05T07:41:25.310] [INFO] default - e[39mSYNCGATEWAY ADMIN USERNAME: admin
e[32m[2022-05-05T07:41:25.311] [INFO] default - e[39mSYNCGATEWAY ADMIN PASSWORD: 123
319ms [I065aa9ae] {26} [INFO] (instance - L:469) Version=2.10.6-njs, Changeset=61e274aae48b6edeecb081afec843dbe31d72790
319ms [I065aa9ae] {26} [INFO] (instance - L:470) Effective connection string: http://0.0.0.0/notifications?enable_errmap=true&enable_tracing=true&client_string=couchnode%2F2.6.12%20(node%2F10.22.0%3B%20v8%2F6.8.275.32-node.58%3B%20ssl%2F1.1.1g). Bucket=notifications
319ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_errmap=true
319ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl enable_tracing=true
319ms [I065aa9ae] {26} [DEBUG] (instance - L:318) Applying initial cntl client_string=couchnode/2.6.12 (node/10.22.0; v8/6.8.275.32-node.58; ssl/1.1.1g)
322ms [I065aa9ae] {26} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
322ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:8091 to initial HTTP bootstrap list
322ms [I065aa9ae] {26} [DEBUG] (instance - L:83) Adding host 0.0.0.0:11210 to initial CCCP bootstrap list
322ms [I065aa9ae] {26} [TRACE] (instance - L:126) Bootstrap hosts loaded (cccp:1, http:1)
e[32m[2022-05-05T07:41:25.535] [INFO] default - e[39mserver started on: 8080 production
e[32m[2022-05-05T07:41:25.535] [INFO] default - e[39mDatabase http://0.0.0.0:4984 http://0.0.0.0:4984
498ms [I065aa9ae] {26} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:81) Preparing providers (this may be called multiple times)
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider CCCP is ENABLED
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider HTTP is ENABLED
498ms [I065aa9ae] {26} [TRACE] (confmon - L:278) Refreshing current cluster map
498ms [I065aa9ae] {26} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:81) Preparing providers (this may be called multiple times)
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider CCCP is ENABLED
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider HTTP is ENABLED
498ms [I065aa9ae] {26} [TRACE] (confmon - L:278) Refreshing current cluster map
498ms [I065aa9ae] {26} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:81) Preparing providers (this may be called multiple times)
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider CCCP is ENABLED
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider HTTP is ENABLED
498ms [I065aa9ae] {26} [TRACE] (confmon - L:278) Refreshing current cluster map
498ms [I065aa9ae] {26} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:81) Preparing providers (this may be called multiple times)
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider CCCP is ENABLED
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider HTTP is ENABLED
498ms [I065aa9ae] {26} [TRACE] (confmon - L:278) Refreshing current cluster map
498ms [I065aa9ae] {26} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:81) Preparing providers (this may be called multiple times)
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider CCCP is ENABLED
498ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider HTTP is ENABLED
498ms [I065aa9ae] {26} [TRACE] (confmon - L:278) Refreshing current cluster map
499ms [I065aa9ae] {26} [INFO] (bootstrap - L:218) Requested network configuration: heuristic
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:81) Preparing providers (this may be called multiple times)
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider CCCP is ENABLED
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:88) Provider HTTP is ENABLED
499ms [I065aa9ae] {26} [TRACE] (confmon - L:278) Refreshing current cluster map
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via CCCP
499ms [I065aa9ae] {26} [INFO] (cccp - L:151) Requesting connection to node 0.0.0.0:11210 for CCCP configuration
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:449) <0.0.0.0:11210> (HE=0x3031760) Creating new connection because none are available in the pool
499ms [I065aa9ae] {26} [TRACE] (lcbio_mgr - L:360) <0.0.0.0:11210> (HE=0x3031760) New pool entry: I=0x2f49a00
499ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:11210> (SOCK=000000003d8f106b) Starting. Timeout=2000000us
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via CCCP
499ms [I065aa9ae] {26} [INFO] (cccp - L:151) Requesting connection to node 0.0.0.0:11210 for CCCP configuration
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:449) <0.0.0.0:11210> (HE=0x2f22650) Creating new connection because none are available in the pool
499ms [I065aa9ae] {26} [TRACE] (lcbio_mgr - L:360) <0.0.0.0:11210> (HE=0x2f22650) New pool entry: I=0x302e160
499ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:11210> (SOCK=000000001ea09bd0) Starting. Timeout=2000000us
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via CCCP
499ms [I065aa9ae] {26} [INFO] (cccp - L:151) Requesting connection to node 0.0.0.0:11210 for CCCP configuration
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:449) <0.0.0.0:11210> (HE=0x2f6cbd0) Creating new connection because none are available in the pool
499ms [I065aa9ae] {26} [TRACE] (lcbio_mgr - L:360) <0.0.0.0:11210> (HE=0x2f6cbd0) New pool entry: I=0x2fccd60
499ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:11210> (SOCK=000000005f62a333) Starting. Timeout=2000000us
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via CCCP
499ms [I065aa9ae] {26} [INFO] (cccp - L:151) Requesting connection to node 0.0.0.0:11210 for CCCP configuration
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:449) <0.0.0.0:11210> (HE=0x3063610) Creating new connection because none are available in the pool
499ms [I065aa9ae] {26} [TRACE] (lcbio_mgr - L:360) <0.0.0.0:11210> (HE=0x3063610) New pool entry: I=0x3029c40
499ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:11210> (SOCK=00000000657b192c) Starting. Timeout=2000000us
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via CCCP
499ms [I065aa9ae] {26} [INFO] (cccp - L:151) Requesting connection to node 0.0.0.0:11210 for CCCP configuration
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:449) <0.0.0.0:11210> (HE=0x2f0b1b0) Creating new connection because none are available in the pool
499ms [I065aa9ae] {26} [TRACE] (lcbio_mgr - L:360) <0.0.0.0:11210> (HE=0x2f0b1b0) New pool entry: I=0x30caa00
499ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:11210> (SOCK=0000000047a94f7b) Starting. Timeout=2000000us
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via CCCP
499ms [I065aa9ae] {26} [INFO] (cccp - L:151) Requesting connection to node 0.0.0.0:11210 for CCCP configuration
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:449) <0.0.0.0:11210> (HE=0x2f17160) Creating new connection because none are available in the pool
499ms [I065aa9ae] {26} [TRACE] (lcbio_mgr - L:360) <0.0.0.0:11210> (HE=0x2f17160) New pool entry: I=0x3063490
499ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:11210> (SOCK=0000000050b960fc) Starting. Timeout=2000000us
499ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:11210> (SOCK=000000003d8f106b) Received completion handler. Status=-111. errno=111 [Connection refused]
499ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:11210> (SOCK=000000001ea09bd0) Received completion handler. Status=-111. errno=111 [Connection refused]
499ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:11210> (SOCK=000000005f62a333) Received completion handler. Status=-111. errno=111 [Connection refused]
499ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:11210> (SOCK=00000000657b192c) Received completion handler. Status=-111. errno=111 [Connection refused]
499ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:11210> (SOCK=0000000047a94f7b) Received completion handler. Status=-111. errno=111 [Connection refused]
499ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:11210> (SOCK=0000000050b960fc) Received completion handler. Status=-111. errno=111 [Connection refused]
499ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
499ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:11210> (SOCK=000000003d8f106b) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:319) <0.0.0.0:11210> (HE=0x3031760) Received result for I=0x2f49a00,C=(nil); E=0x10
499ms [I065aa9ae] {26} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
499ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'CCCP' failed
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:220) Will try next provider in 0us
499ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
499ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:11210> (SOCK=000000001ea09bd0) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:319) <0.0.0.0:11210> (HE=0x2f22650) Received result for I=0x302e160,C=(nil); E=0x10
499ms [I065aa9ae] {26} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
499ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'CCCP' failed
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:220) Will try next provider in 0us
499ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
499ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:11210> (SOCK=000000005f62a333) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:319) <0.0.0.0:11210> (HE=0x2f6cbd0) Received result for I=0x2fccd60,C=(nil); E=0x10
499ms [I065aa9ae] {26} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
499ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'CCCP' failed
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:220) Will try next provider in 0us
499ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
499ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:11210> (SOCK=00000000657b192c) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:319) <0.0.0.0:11210> (HE=0x3063610) Received result for I=0x3029c40,C=(nil); E=0x10
499ms [I065aa9ae] {26} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
499ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'CCCP' failed
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:220) Will try next provider in 0us
499ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
499ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:11210> (SOCK=0000000047a94f7b) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:319) <0.0.0.0:11210> (HE=0x2f0b1b0) Received result for I=0x30caa00,C=(nil); E=0x10
499ms [I065aa9ae] {26} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
499ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'CCCP' failed
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:220) Will try next provider in 0us
499ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
499ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:11210> (SOCK=0000000050b960fc) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
499ms [I065aa9ae] {26} [DEBUG] (lcbio_mgr - L:319) <0.0.0.0:11210> (HE=0x2f17160) Received result for I=0x3063490,C=(nil); E=0x10
499ms [I065aa9ae] {26} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_NETWORK_ERROR (0x10)
499ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'CCCP' failed
499ms [I065aa9ae] {26} [DEBUG] (confmon - L:220) Will try next provider in 0us
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via HTTP
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via HTTP
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via HTTP
499ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via HTTP
500ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via HTTP
500ms [I065aa9ae] {26} [TRACE] (confmon - L:265) Attempting to retrieve cluster map via HTTP
500ms [I065aa9ae] {26} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x2f60e70
500ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:8091> (SOCK=000000003cc48302) Starting. Timeout=2000000us
500ms [I065aa9ae] {26} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x30315c0
500ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:8091> (SOCK=0000000048c48685) Starting. Timeout=2000000us
500ms [I065aa9ae] {26} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x2f6bb30
500ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:8091> (SOCK=000000004eba20be) Starting. Timeout=2000000us
500ms [I065aa9ae] {26} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x2f64e70
500ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:8091> (SOCK=00000000477077e2) Starting. Timeout=2000000us
500ms [I065aa9ae] {26} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x2f68bb0
500ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:8091> (SOCK=00000000483f1273) Starting. Timeout=2000000us
500ms [I065aa9ae] {26} [TRACE] (htconfig - L:347) Starting HTTP Configuration Provider 0x30d8640
500ms [I065aa9ae] {26} [INFO] (connection - L:474) <0.0.0.0:8091> (SOCK=000000004ed030e7) Starting. Timeout=2000000us
500ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:8091> (SOCK=000000003cc48302) Received completion handler. Status=-111. errno=111 [Connection refused]
500ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:8091> (SOCK=0000000048c48685) Received completion handler. Status=-111. errno=111 [Connection refused]
500ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:8091> (SOCK=000000004eba20be) Received completion handler. Status=-111. errno=111 [Connection refused]
500ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:8091> (SOCK=00000000477077e2) Received completion handler. Status=-111. errno=111 [Connection refused]
500ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:8091> (SOCK=00000000483f1273) Received completion handler. Status=-111. errno=111 [Connection refused]
500ms [I065aa9ae] {26} [TRACE] (connection - L:367) <0.0.0.0:8091> (SOCK=000000004ed030e7) Received completion handler. Status=-111. errno=111 [Connection refused]
500ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
500ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:8091> (SOCK=000000003cc48302) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
500ms [I065aa9ae] {26} [ERROR] (htconfig - L:301) Connection to REST API failed with LCB_NETWORK_ERROR (0x10) (os errno = 111)
500ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'HTTP' failed
500ms [I065aa9ae] {26} [TRACE] (confmon - L:225) Maximum provider reached. Resetting index
500ms [I065aa9ae] {26} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x3000520. Error=LCB_NETWORK_ERROR (0x10), Message=No more bootstrap providers remain
e[91m[2022-05-05T07:41:25.539] [ERROR] default - e[39morganization bucket initialize error -  { [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[36m[2022-05-05T07:41:25.539] [DEBUG] default - e[39m{ [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39morganization bucket on error event - CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information
501ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
501ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:8091> (SOCK=0000000048c48685) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
501ms [I065aa9ae] {26} [ERROR] (htconfig - L:301) Connection to REST API failed with LCB_NETWORK_ERROR (0x10) (os errno = 111)
501ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'HTTP' failed
501ms [I065aa9ae] {26} [TRACE] (confmon - L:225) Maximum provider reached. Resetting index
501ms [I065aa9ae] {26} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x3030ac0. Error=LCB_NETWORK_ERROR (0x10), Message=No more bootstrap providers remain
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39muser-config bucket initialize error -  { [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[36m[2022-05-05T07:41:25.540] [DEBUG] default - e[39m{ [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39muser-config bucket on error event - CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information
501ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
501ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:8091> (SOCK=000000004eba20be) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
501ms [I065aa9ae] {26} [ERROR] (htconfig - L:301) Connection to REST API failed with LCB_NETWORK_ERROR (0x10) (os errno = 111)
501ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'HTTP' failed
501ms [I065aa9ae] {26} [TRACE] (confmon - L:225) Maximum provider reached. Resetting index
501ms [I065aa9ae] {26} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x3031a00. Error=LCB_NETWORK_ERROR (0x10), Message=No more bootstrap providers remain
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39muser-projects bucket initialize error -  { [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[36m[2022-05-05T07:41:25.540] [DEBUG] default - e[39m{ [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39muser-projects bucket on error event - CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information
501ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
501ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:8091> (SOCK=00000000477077e2) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
501ms [I065aa9ae] {26} [ERROR] (htconfig - L:301) Connection to REST API failed with LCB_NETWORK_ERROR (0x10) (os errno = 111)
501ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'HTTP' failed
501ms [I065aa9ae] {26} [TRACE] (confmon - L:225) Maximum provider reached. Resetting index
501ms [I065aa9ae] {26} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x3012ac0. Error=LCB_NETWORK_ERROR (0x10), Message=No more bootstrap providers remain
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39mevents bucket initialize error -  { [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[36m[2022-05-05T07:41:25.540] [DEBUG] default - e[39m{ [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information]
  message:
   'Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information',
  code: 16 }
e[91m[2022-05-05T07:41:25.540] [ERROR] default - e[39mevents bucket on error event - CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information
501ms [I065aa9ae] {26} [WARN] (lcbio - L:112) Translating errno=111, lcb=0x2c to NETWORK_ERROR
501ms [I065aa9ae] {26} [ERROR] (connection - L:156) <0.0.0.0:8091> (SOCK=00000000483f1273) Failed to establish connection: LCB_NETWORK_ERROR (0x10), os errno=111
501ms [I065aa9ae] {26} [ERROR] (htconfig - L:301) Connection to REST API failed with LCB_NETWORK_ERROR (0x10) (os errno = 111)
501ms [I065aa9ae] {26} [INFO] (confmon - L:179) Provider 'HTTP' failed
501ms [I065aa9ae] {26} [TRACE] (confmon - L:225) Maximum provider reached. Resetting index
501ms [I065aa9ae] {26} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x2f67f80. Error=LCB_NETWORK_ERROR (0x10), Message=No more bootstrap providers remain
2

Hi,

So some of this is going to come down to your “connection string” or your configuration to connect to Couchbase Server. Is this a local development machine? Are you using something like Docker? If a local development machine what is your connection config information - i.e. what is your connection string you are using to connect to your local instance of Couchbase Server?

-Aaron

3

Hi Aaron,

Thanks for a quick reply. The node.js server is not on the local development machine and is deployed in a Docker container in a Kubernetes cluster on cloud.
For the connection, I am just passing the IP address of the server (eg. 0.0.0.0) while creating a new Cluster object using the SDK and the name of the bucket to the openBucket method of the cluster object.

4

@prasadb, so trying to make sure I understand this correctly. So both your database server and your webserver are deployed into K8S in some cloud provider? If this statement is true, I’m assuming these are two different pods in K8S. Is this correct?

If this is the case, then a few things:

  1. K8S usually runs internal DNS, and you would want to use the machine name, not the IP address on your web backend server, to talk to the database. Not using an IP is even more critical when you have more than one node in your cluster, as if that pod fails, your web server has no other servers to fall back on.

  2. You will want to review your configuration to ensure the K8S pods can talk to each other. Your network isn’t connecting, and that could be the connection string IP address, the network configuration in K8S where the pods can’t talk to each other, the proper ports for CB Server aren’t exposed, or several other problems I’m not thinking of. I believe it’s one of these due to this error:

e[36m[2022-05-05T07:41:25.539] [DEBUG] default - e[39m{ [CouchbaseError: Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via detailed_errcodes in the connection string) and/or enable logging to get more information]
message:
‘Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via detailed_errcodes in the connection string) and/or enable logging to get more information’,
code: 16 }

If this isn’t the case, can you clarify the infrastructure better? It’s a network error in the log file that you posted. I need to make sure we know everything about the network to help us better help you.

Thanks
Aaron

5

Hi Aaron, sorry for not being clear about the deployment of the database server.

Below is the architecture:

  1. Node.js is deployed as a Docker container in Azure K8S cluster.
  2. Couchbase server is deployed on VM on a different network than K8S cluster.
  3. Couchbase VM has a load balancer with load balancing rules for 8091, 8092, 8093, 11210, 4984, 4985.
  4. Node.js is connecting to Couchbase server using private endpoint connected to load balancer.
  5. Versions:
    a. Node.js Couchbase SDK - 2.6.12
    b. Couchbase Server - Community Edition 6.5.1 build 6299

A few observations:

  1. The issue happens randomly and everything works fine after few restarts of the Node.js K8S pod.
  2. Today I faced the issue again after 12 days of everything working fine. Below are the debugging steps I tried.
    a. Restarted other K8S pods connecting to Couchbase server using Java SDK and Python SDK. They were able to connect to Couchbase server successfully. These K8S pods were also in the same VM as that of the Node.js K8S pod.
    b. I checked all the ports required for connecting to Couchbase server.
    c. Restarted the Couchbase server. But, the issue was not resolved.
    d. I scaled down Node.js K8S pod for 5 minutes and then scaled it up. Everything worked fine after that.

Thanks
Prasad