Sync gateway raised the error "WS: c:[5abf1836] ERROR decompressing frame: inputLen=4090, remaining=0, output=0, error=unexpected EOF" in k8s

Mobile Sync Gateway
1

Hello All,

I am deploying the couchbase server (6.0 community edition) and sync gateway (2.6.1 community edition) to k8s environment, the couchbase server and sync gateway are running in the same docker container and behind a web server - Traefik . But I am facing the issue regarding “unexpected EOF” in sync gateway when saving data (Json format). I am able to save some specific data (json format) without issue but there is only one case it caused this error.

I didn’t see this issue when I deployed couchbase server and sync gateway to AWS ECS. Everything is working fine in this environment.

So I am wondering whether anyone is facing the same issue.
Any advice are appreciated!

Below is the error log which I extracted from sg_info.log
2020-07-30T20:41:56.847Z [INF] HTTP: #893: GET /offline/_blipsync (as GUEST)
2020-07-30T20:41:56.847Z [INF] HTTP+: #893: --> 101 [117966a2] Upgraded to BLIP+WebSocket protocol (as GUEST) (0.0 ms)
2020-07-30T20:41:56.847Z [INF] WS: c:[117966a2] Start BLIP/Websocket handler
2020-07-30T20:41:56.944Z [INF] SyncMsg: c:[117966a2] #1: Type:getCheckpoint Client:cp-ISneQUdMy4rgoNEoIPSaCA8b8/8=
2020-07-30T20:41:57.032Z [INF] SyncMsg: c:[117966a2] #2: Type:proposeChanges #Changes: 1
2020-07-30T20:41:57.069Z [INF] SyncMsg: c:[117966a2] #3: Type:subChanges Since:35 Continuous:true Filter:sync_gateway/bychannel Channels:ABC
2020-07-30T20:41:57.069Z [INF] Sync: c:[117966a2] Sending changes since 35
2020-07-30T20:41:57.069Z [INF] Changes: c:[117966a2] MultiChangesFeed(channels: {ABC}, options: {Since:35 Limit:0 Conflicts:false IncludeDocs:false Wait:true Continuous:true Terminator:0xc000e6ca20 HeartbeatMs:0 TimeoutMs:0 ActiveOnly:false Ctx:context.Background.WithValue(base.LogContextKey{}, base.LogContext{CorrelationID:"#893"}).WithValue(base.LogContextKey{}, base.LogContext{CorrelationID:"[117966a2]"})}) …
2020-07-30T20:41:57.069Z [INF] Sync: c:[117966a2] Sent all changes to client
2020-07-30T20:41:57.250Z [INF] WS: c:[117966a2] ERROR decompressing frame: inputLen=4090, remaining=0, output=0, error=unexpected EOF

2020-07-30T20:41:57.251Z [INF] WS: c:[117966a2] Error receiving frame MSG#4~: unexpected EOF. Raw frame = <04489cdb5bb2f5385205e0622a3d0259bec86236b264477404d10434f0c21c1833dfda05ef04fd507dfe73d9965299eb924affd73ffcef4afef5fd8f3ffebafef87f3dfef7c7fff705fcf1f7f75ffefdfddb7cffa8db1f7ffbe7f9cf7ffbfee9aff3dffefec7f6c77ffee55ff3b3bfffdb5ffef12f57bdb66b5ff7789fb94a7fb7fd3d8eab6ffb751faf9fedd73ad67e7de7d7be32c7766c735edfdbdfe7bab7adafcfe2b6bacf6b6b759579ecd79c7dbe5b7fe7768cab8f7b2bc5f68a35f97ea9e7ea6bbdcfd1d6acfb1089ef5bd77bcd368ebd97329efb2ddb558ff5ad75551bd9eb715ad0b3ddefbdaa3d1ed67a3ed736b6eb7a8eed29ef38e6f35e7bdbde6ddc79c8fecdf97c9e7bdcfdb9e7dcb7b57df7b0e6be97f95cefd79febbaaa953febbdaee3bd9f7ddfe777f563de47e9abd5a3d4713ecbf6c6e9c7db96b338adbbaeabd46edf75ddf57a9fdadeab974d34fa5cadbcd7b1eef379cef28d752d7b1f63bebd8e67adfdb5f3e312a4f13cad94b9de6e7ff33dc7facaf3f4d25b2f12e199f7be3bdd6bbfc77efbe0b65d8edc6f7fdb7bda775bfbbc0577dfd735cab4fcd112b3fdb499a7ccf3fac618e599e5d9379fd6fcf1f11c9e77f7f71be7758ad4e8e37b7a7b6739dbf176cfbf87259c8f783eedddee04fc9dcf3a76ab5d4232befb3ab6b39fe53d9fafcdf39487db78976f38a355ab939ef3f4bcfd7bdbf7bc4e748ad031e4563ffb588fb00a472d677d2e797ff46ff8decc077cd77d3b9871fae2f27869b1d7fd6db3f756bffe9664d757fb55f3a8bb3bd9d6afc31ae6373e09f4ca87e32af751cfef3dcbb8d63a9ecdceded747977d94f2d8c0d7d4dd285f1bad5965cea7dff6b8de7a9fef14c1bdb67ad7b73d43869f52e1befb359ef79bb25490df350eb1d95ee932f752fcbb5ecf2e731ee7f33d6b5bd2b03eb52aaffb7bf7fd19ef1ce77e28d577dc52a89d6faba71dbe8f7aebad48d17d7ccebcad6d2bd7d3e7b6fab01d69738e611b4b6a1c625305f6b9ca38db7e3efbecf5aa53e48eab7e524254d63c85f55616e3295f5ff3179d5b86d74d727c0e78acf3db37215fc748293f52fcd95b7954c52795ded14189225566f7d3ea7c24e4db2c744fc1dd162db58e5985b15c40e914e4bd9dfda8f265abfe71fc890cbb85e6f077e7feddef2ea58efd68738dfb38f7a71ef67c9e4f53180021e7337a1baf806ce7aa223c1fb9b29f01a239da7baeb2ae47ca3be06d1c3d61b0dcf7567ef5695777a2fe5ac27790e49fef7bbca53804e1ecc733f7f7de7b9df5a84ee69203f77aa7dd6d6b7ce7795a2f907b1cc6dcf671ac76abfb79963624717b1fdb5655739b8e0d2ccdebdd5f4b3a7f187ecef75ac078be763ec6f7aead6f76b3a4ef079f170455065f3dcfeff91efb2eb3780cacf0c14f55f0eb52acef5e7bff1c4fb3ec63545f2ab96dff9e6b3f7795a5e8eb7394b7ddcf77431d0966b3d7bc8efd75c2fd09533cb6d4606c159bdabe59eeab9f42e888a79ff5ef0061e7230ddfaf02e70220e64852b77642997296e34e162a9c5d72416b8b9671bf746e4eee798f55bf5dacca37b76b41e07e7fa521937e9dc775fb4367ef73e638bed69eaded03b07ff04a16cdd9e6714fd1e88ab52d0971aceb93c0ef0250bf8c6bedbb171484e032a55d6fddee94f4e7b83cefeaeb7eb333e961a5972780e1b98ea39dc06c930cf52980a6adf57c7bdb9f54eb55e0f4f3dc159c3b9ea31428026bea2643c4edb891885a80e9f37d6fc4aa1e9b0a16159966030016aa7ff3dc9e13f01c2af61548a0b24e356dcd67ddc6aa3de77a9d00efae37ba7fdefd6e6193bbf480473b6b3d6b5f2a0bd3d50f19dbf16713c77dbfb5d773dfcf81b83ec0b6b5b203278baf20b4e3b9da07009adbd301850afa4ebf3a8afcde24bda21a09c937dbfae4fb750f15b5cdfa9c873469b2f06ecaaaaca16600520fa70385779fad2fa5efa3de321d3eaef3334bdaef3ac455d6bc32ef79576fd7ba93aaf61aacf9d5260c59c0ee6a5fbb10ebb57fca74bfea6a139517f43f0510f9f44de92810004d2258d1794ff5fc94eee01a68ebfb195ec2b45f6ff6ba144ff5fbcf51c7126c07eb94f6d1ef774b896160e01dc4dbfcde7428d863b4af1f6ba3a81a4a5ae5dc3fd1878032515c15013910842a7d02e54ec17ccd116d2203258eb99d05a710143277dff15ab1e7797ce7559e764ac469c94170950f3fd0ecc4f51dacd5670c8782303e1878cd8dbaa9aaf4d92fdf7ebed51f4bdad0c156f0a482f9a0f37183a6078cce626765c9601f0907ea189068dfa1e48dfca720c0e1b1bd2bec8eb8ced4d9d6d4b8df40109dd43c140b607d901ae1422311011005e60caaa3546045fbb5e391d59f48d80b525ecf6a5d86bc8b421282fd939e8bf080ef7b73c84300f7e77854f7eea470d6ab341b36752227ddf5101fc4c86dc74e2730fc40ec6b5cdffc92d711768eab503d96788680252846916b176abe9eb38d67a8ec731cafa7a907740371b63ef67e5cdf05adf0fd73f41e2a1a517d25aa01b4be94e45b7fac9e429c109a4e23bc6ab1d38fb01a74d9532a45fd0c9c462f648f3e014bdf34d4cae3f6e83622b84477dcaf7d5590ddc65ed0d2ae1ee46611dc2bcaf4b48ba2966fc2f3226414cf076840f23ca0c5fd53adfe601c7713e7e3a00d12a64dfa56ca45414125587c8f51c9bc420f7ee88da61070acf96d470164573b41dc96bcb8a82d24d991e54eeee0c0f390e610bed0541400ee6e091712bc8899624d13f739d9afa0ebc85d6c54aeb3d176430e2c3c74fa2520062c65c1d8479d341aa9de9e13e5b10dccca393cb66759a2e371e89000553bd7044784c8905e51508e106891c8c4a4fc80f2cccbb6d751ef870e6da5091f3a7c6e689ee26ccaa2d448c042a154a16d0e5158717e44d3a0af8006fef8c0e9454203417a5402d4c84020de7f719b11c90cd4abc85f625a84d006c257587c463da4563c1450ea7e9d036841d8f19edd9e3007f9398851b411d426706e0f842772a7d354919f4a00531d899d452a3404144af812249ff4c17252d2df78626c1321c71151daca88b9023177900356f37d89ea74828b46c2970e8923f4cd6b07ce83b2a47b6ba08a2c43b70b13071591199300767b7bfa48d43000d254cef3dbf1a6206de10510264c0cc693bf27d13b3482dc0560726ce71debe07c8f6380f16b57513d64ecd8f720e2fd0ccceab9befccec0db1be8530d740b183c1636a69a7c874520923d0e374328b1de1672bf4b8ccd7b124701a48392e3178e510e454141b256fe13e33294f013f6dc7a785b7d0727992031a017b00b7188400501952bacc3426cfa87a667a3f9e8710ae4ebd6e2c95ffcad272c2a1a2c5d800a0c5d306f722e4acce9f3c0907e8ffc3d9deb175b0cc06d11af5ff50e49d676011292ec7880e05d6970e47a9182633fd896c9eb38a2c73ee4b40a8cd3edcee12e95d968f2c101ef24b28f89c150c9abb1bba48127c52ffa7dee4456625092d386a268eeea30901dfff6a1e1835c0d6a9e2d9001b9caa804f5887d7d596e4a8f56e5cdbf6b9b9fca0305984d5ddd6431429989fa0f4ae25b1839c1c32bbc3299fc53203b481e37997ea9d2f3e0e7e407bc5737f96f95e64c4fcde7c23bf52e957c088cd8229d2fab54935f8d8c63f62e31e100265c632177eb4c03606d7411edae3093f5443cbe5585fcc614258abb4414906828fe012eeb0d3fac91ae458b7f8c0322a82c0f3f390412cda7b004df9aed965ea8e4f4ade326d9544b932a61868d82afdce15cea8c2a193c4bc4e4c7bc45b1a6be9b335b695348f9b62d3a62075a3d315dabcb26c0c99ff08aa8f1f1e0369fbbe114ff3ba45ea237f1ce4147531d14bbcc76403e00d0de04c1fd0ae145ead5ba2bcd8155b8864f08680492d74f17d704e3e9882eab3fe81db35b643d6510337b83cc5d26d04d64633c26a1079a36f8a816ae73c3c264c16243d57c4365008aed94f1b11d9ddfe27e41bbc751f436f412b33d5f6539f52dfb942b89eb192c75e420ac210874c35bab754cfc1dbdf819e5403a338e231d9e5a4a943bef8616d722aee5ae7c79df45fbdbe83b19248886d7f691037d932e49b4f7a810195260854a1b853a04a6ff2c1393d7484ff9b9780e04f203a41af7591a0fb485f5d500d77c04951688c179dfe7d8a16f7a41fd27862e1fba4ec7000b61f38ac8df72a47c01855be4df113725032f8c7c3a0deadb2fbd373556b86f06636be2c533b3f0d0ede7242f75b50e01f900141878b9139831a91c5ab3c868347ad175d4a7ac9b82883e089c464f46e3bde1e0bd4676f9f88b08d844a03f54abf3bea52aa14deaa8ca45e6011e8bb90ae6e76f9c85454f45c4be1e16c337c0c670f345f42160872061c84776bc46be10a8f6c35e3c11c77bb13c894a39ef07c10f8686347cd30ca1b9cf9e6782dfb2722cfb696d321e7ef42fb6e0851feb3da9fc2f913f81e696ad280b062ebef358a5a054ec7e00b31bc09fc903cfe6685561f437abd42fc4f1f319973a2491a5007495bec1fdef3e77c470c46f36003830c8872b9509e5ded41e6221b329a50f529d445fcb19bee7cd841e51c1628889eb76324328823b05294fb91b27445249fe29130a4002ff1f3113a9f0aa1e0a52498f4d443f1fc6a459ce5bae48625086f464a3f2c273658a94ff0b207e24d636d3506adc32956ec997bdf7747c136e30dba50a9aec76f7451ead200b938c2018d25b76814b722c569aec26377eac621dea7eb331929ec4a45eef3caea2bfabe3d9974322cbf6bda5834b42a1e62be292a6c68711d58ef85edb4797a23ffa919be252c957680430a7f251528fdd7d617ae8e7d727f531a387d92bb00c53fde7018961a54593c948695be4b7e0574e884e842a4448b938dcc706cbbca3f75e477f8e9342e9fbbec814eab1e20c4a8f512ca74d310e638fa46f6ad22a3078e2d4ad74bd60ab8b0eb7c18803ebd95fa4e070e01bcbc9b493f75b6abb0bf3ad6eb7e344c1a458023a7d18fb57a23d24708dba705210679c967afb38d638501c887dd3db22b1c028e3e53b747e2847eede25fd1e2138fde659a3774a6acfd20b6425153c8189a423110a4e876b16a036d972f27e5fdcd1715d7208c0581e1ba08a63e84f92269df5aafa0711f7279102b2d1360209a40d417962f9a9572c3481f071c5a34b59ec72d234517e9c5a52364df7453928c2884b08586c9e9f2285303269bbc128a501ff7ced64d42083c5abb1d91bc0388937e60332ece50c5287ef3917a736a8def18df3178c6bb45bb2133c4ee3cef28a349534507c3f021ea72366faf024cd4dfbaaa0f2317a6f3de839fa62c68d02dae7bea357cf2103a9b23d4db56b79eab1e71fea30d0e00b064c2562090a8ca1cc091d14f722b026f320d51504d0a508adca69a1a36c66fb0ed6eff09c179c397cb2f4fdfac5bbc9614a89069d689bd2c2b5274613b4e37b730391be65a3f9e0d4e36150d281b31f15b66c6b43e9490898b5a8ba30c6d89c123382c74485c16931df753de7cea4347426e27ee9941bd709e5a1e95833e98a2715f4afbdd0700461c63411e374a12ac55692723db0ae47397c0e436aa6410db83e7a455ecaa18b2fa01a6017f61910f6d8ca96ee4d8a96bf3d9ff4412402976b9bfba972d9794e02bb9efebf2b78facd039e2e7defef4eae330c0fe13d5ebebdd7509a3c80fb940c0e4128874df224170d2418734fec3e82bd0a401ad92fb644bedb9be6c01647f1a5cbd7aff44f9c>
2020-07-30T20:41:57.251Z [INF] WS: c:[117966a2] Error: parseLoop closing socket due to error: unexpected EOF
2020-07-30T20:41:57.251Z [INF] WS: c:[117966a2] BLIP/Websocket Handler exited: unexpected EOF
2020-07-30T20:41:57.251Z [INF] HTTP: c:[117966a2] #893:     --> BLIP+WebSocket connection error: unexpected EOF
2020-07-30T20:41:57.251Z [INF] HTTP: c:[117966a2] #893:    --> BLIP+WebSocket connection closed
2020-07-30T20:41:57.251Z [INF] Changes: c:[117966a2] MultiChangesFeed done
2

Sounds like an issue with the load balancer. I would recommend reviewing the Traefif guide and apply these general guidelines

On unrelated note,

the couchbase server and sync gateway are running in the same docker container and behind a web server - Traefik

Same docker container ??? So you created a custom image with couchbase server and sync gateway ? Or did you mean “same pod” - even thats not a recommended practice to have sync gateway and server on same pod. They have different sizing and scaling requirements

3

Thanks Priya for the answer.

I setup in the same pod, I don’t need the scaling for couchbase in my case, I just need one instance for couchbase sync gateway and couchbase server.

I will take a look into Traefik and loadbalancer to see if I can solve this issue. Will let you know soon

4

Curious- why are you using Kubernetes ? It seems like a pretty trivial one pod deployment of server and sync gateway.

5

Hi Priya,

Sorry for late responding.

Curious- why are you using Kubernetes ? It seems like a pretty trivial one pod deployment of server and sync gateway.

Yes, I am trying to deploy couchbase server/sync gateway to Kubernetes to see if these services are working fine in this environment.
This is a trivial, if it is working fine then I will apply it. Any recommendations regarding this deployment are really appreciated!

BTW, I checked the LB keep-alive timeout then it is already set to 1 hour. One more thing I need to do is changing the keep-alive timeout of Traefik but this is not recommended due to impact to other applications.
Do we have an alternative way to change the keep-alive timeout in Traefik instead of updating Traefik configuration directly?

Or Is it possible to change the heartbeat timeout (in couchbase lite/sync gateway)?

Tuan

6

Recommendation to deploy Couchbase Server and Sync Gateway in separate pods and preferably scheduled on separate nodes .

Regarding EoF issue, did you try accessing Sync Gateway via some other load balancer

7

Thanks Priya for recommendation, I will check on this…

Yes, I did try to deploy in AWS ECS using AWS load balancer and it was working fine.

Regarding EOF issue, I tried to reduce the size of payload per request then this issue gone.
There is a similar issue which described here: Websocket response are not complete · Issue #4446 · traefik/traefik · GitHub , it seems like the issue related to Websocket.

Any comments on that issue link?