Strange issue: Could not decode snappy-compressed value

I’m running into this strange issue - pretty consistently right now…
Soc I have a document where I get an exception when I try to Get(doc id) - this is the snippet where I get the first log line - but not the second:

Util.info("key=" + key + ", docid=" + getDocId(key));
JsonDocument doc = getDb().get(getDocId(key));
Util.info("Doc. loaded ");

The exception is:

Caused by: com.couchbase.client.core.CouchbaseException: java.lang.RuntimeException: Could not decode snappy-compressed value.
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:369)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:811)
Caused by: java.lang.RuntimeException: Could not decode snappy-compressed value.
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.handleSnappyDecompression(KeyValueHandler.java:397)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:949)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:132)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:338)
	... 33 more
Caused by: com.couchbase.client.deps.org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting at 1
	at com.couchbase.client.deps.org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
	at com.couchbase.client.deps.org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:47)
	at com.couchbase.client.deps.org.iq80.snappy.Snappy.uncompress(Snappy.java:85)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.handleSnappyDecompression(KeyValueHandler.java:395)
	... 36 more

The document that fails seems pretty straight forward to me:

{
  "catchphoto": false,
  "datetaken": "2015-12-08T13:45:47+0100",
  "filepath": "/photo/ab/ab720a1f-dbd0-4ac5-9fef-a9e86ac32b3b.jpg",
  "height": 125,
  "ispublic": false,
  "key": "740B51E6D8BD1BC5C1257F1500461C63",
  "revisioninfo": {
    "created": "2015-12-08T13:45:48+0100",
    "createdby": "John Dalsgaard/2124DEFEC111BA8FC1257ED20034B387/Fangst",
    "modifiedcount": 0
  },
  "size": 3688,
  "thumbnailfilepath": "/photo/ab/ab720a1f-dbd0-4ac5-9fef-a9e86ac32b3b-tn.jpg",
  "thumbnailheight": 100,
  "thumbnailsize": 2703,
  "thumbnailwidth": 120,
  "type": "Photo",
  "width": 150
}

And the meta data:

{
  "meta": {
    "id": "Photo:740B51E6D8BD1BC5C1257F1500461C63",
    "rev": "2-159c2bd3020200000000000002000000",
    "expiration": 0,
    "flags": 33554432,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "1-0314fe729a6efb9e7ac7896e486b640c",
      "sequence": 322931,
      "recent_sequences": [
        322931
      ],
      "history": {
        "revs": [
          "1-0314fe729a6efb9e7ac7896e486b640c"
        ],
        "parents": [
          -1
        ],
        "channels": [
          null
        ]
      },
      "cas": "0x00000202d32b9c15",
      "value_crc32c": "0x5a509151",
      "time_saved": "2019-05-06T20:35:56.405028379+02:00"
    }
  }
}

… and it also happens with other similar “Photo” docs (that are loaded on the first page of the web app). So basically, the entire web app is not functional at the moment… (it is in a demo environment - but of course it is worrying… - can this also happen in production?)

Where can I start troubleshooting this?

Environment:
Couchbase server: Community Edition 6.0.0 build 1693 ‧ IPv4
Java 1.8 using the couchbase client SDK 2.7.9

Not sure if this matters - but this environment also has a Sync. Gateway: (Couchbase Sync Gateway/2.6.0(127;b4c828d) CE). We use this environment for developing a new version of our app that uses Coucbase and Sync Gateway :slight_smile:

Rebooting all servers involved did not make any difference…

Noone knows anything about this???

I get more errors when this happens - not sure which one is the egg and which one is the chicken…

However, this may help in finding out:

 Image.get: ERROR - java.util.concurrent.TimeoutException: {"b":"data","r":"192.168.42.211:11210","s":"kv","c":"53DDE94D1C664D37/000000003EB92E84","t":2500000,"i":"0x5e9","l":"192.168.42.226:55326"}
 java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"data","r":"192.168.42.211:11210","s":"kv","c":"53DDE94D1C664D37/000000003EB92E84","t":2500000,"i":"0x5e9","l":"192.168.42.226:55326"}
 	at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
 	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
 	at rx.observables.BlockingObservable.singleOrDefault(BlockingObservable.java:372)
 	at com.couchbase.client.java.CouchbaseBucket.get(CouchbaseBucket.java:139)
 	at com.couchbase.client.java.CouchbaseBucket.get(CouchbaseBucket.java:134)
 	at dk.dtu.aqua.catchlog.dao.CouchbasePhotoDAO.getByteImage(CouchbasePhotoDAO.java:165)
 	at dk.dtu.aqua.catchlog.dao.facade.PhotoCRUDFacade.getImage(PhotoCRUDFacade.java:156)
 	at dk.dtu.aqua.catchlog.service.Image.processGetRequest(Image.java:65)
 	at dk.dtu.aqua.catchlog.service.Image.get(Image.java:76)
 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
 	at java.lang.reflect.Method.invoke(Method.java:508)
 	at com.ibm.jscript.types.JavaAccessObject.call(JavaAccessObject.java:322)
 	at com.ibm.jscript.types.FBSObject.call(FBSObject.java:161)
 	at com.ibm.jscript.ASTTree.ASTCall.interpret(ASTCall.java:197)
 	at com.ibm.jscript.ASTTree.ASTProgram.interpret(ASTProgram.java:119)
 	at com.ibm.jscript.ASTTree.ASTProgram.interpretEx(ASTProgram.java:139)
 	at com.ibm.jscript.JSExpression._interpretExpression(JSExpression.java:435)
 	at com.ibm.jscript.JSExpression.access$1(JSExpression.java:424)
 	at com.ibm.jscript.JSExpression$2.run(JSExpression.java:414)
 	at java.security.AccessController.doPrivileged(AccessController.java:730)
 	at com.ibm.jscript.JSExpression.interpretExpression(JSExpression.java:410)
 	at com.ibm.jscript.JSExpression.evaluateValue(JSExpression.java:251)
 	at com.ibm.jscript.JSExpression.evaluateValue(JSExpression.java:234)
 	at com.ibm.xsp.javascript.JavaScriptInterpreter.interpret(JavaScriptInterpreter.java:222)
 	at com.ibm.xsp.binding.javascript.JavaScriptMethodBinding.invoke(JavaScriptMethodBinding.java:111)
 	at com.ibm.xsp.component.UIViewRootEx.invokePhaseMethodBinding(UIViewRootEx.java:1735)
 	at com.ibm.xsp.controller.FacesControllerImpl.invokePhaseMethodBinding(FacesControllerImpl.java:450)
 	at com.ibm.xsp.controller.FacesControllerImpl.access$0(FacesControllerImpl.java:444)
 	at com.ibm.xsp.controller.FacesControllerImpl$ViewPhaseListener.beforePhase(FacesControllerImpl.java:533)
 	at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:197)
 	at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:120)
 	at com.ibm.xsp.controller.FacesControllerImpl.render(FacesControllerImpl.java:270)
 	at com.ibm.xsp.webapp.FacesServlet.serviceView(FacesServlet.java:260)
 	at com.ibm.xsp.webapp.FacesServletEx.serviceView(FacesServletEx.java:157)
 	at com.ibm.xsp.webapp.FacesServlet.service(FacesServlet.java:159)
 	at com.ibm.xsp.webapp.FacesServletEx.service(FacesServletEx.java:138)
 	at com.ibm.xsp.webapp.DesignerFacesServlet.service(DesignerFacesServlet.java:103)
 	at com.ibm.designer.runtime.domino.adapter.ComponentModule.invokeServlet(ComponentModule.java:588)
 	at com.ibm.domino.xsp.module.nsf.NSFComponentModule.invokeServlet(NSFComponentModule.java:1335)
 	at com.ibm.designer.runtime.domino.adapter.ComponentModule$AdapterInvoker.invokeServlet(ComponentModule.java:865)
 	at com.ibm.designer.runtime.domino.adapter.ComponentModule$ServletInvoker.doService(ComponentModule.java:808)
 	at com.ibm.designer.runtime.domino.adapter.ComponentModule.doService(ComponentModule.java:577)
 	at com.ibm.domino.xsp.module.nsf.NSFComponentModule.doService(NSFComponentModule.java:1319)
 	at com.ibm.domino.xsp.module.nsf.NSFService.doServiceInternal(NSFService.java:662)
 	at com.ibm.domino.xsp.module.nsf.NSFService.doService(NSFService.java:482)
 	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:357)
 	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:313)
 	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)
 Caused by: 
 java.util.concurrent.TimeoutException: {"b":"data","r":"192.168.42.211:11210","s":"kv","c":"53DDE94D1C664D37/000000003EB92E84","t":2500000,"i":"0x5e9","l":"192.168.42.226:55326"}
 	at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131)
 	at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127)
 	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
 	at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166)
 	at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191)
 	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
 	at java.util.concurrent.FutureTask.run(FutureTask.java:277)
 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:191)
 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 	at java.lang.Thread.run(Thread.java:811)

And the line of my code that gets the exception
dk.dtu.aqua.catchlog.dao.CouchbasePhotoDAO.getByteImage(CouchbasePhotoDAO.java:165)
Which is in this function:

	public byte[] getByteImage(String key, boolean thumbNail) {
		if (StringUtil.isNotEmpty(key)) {
    Line 165 -->	JsonDocument imageDoc = getDb().get(getImageDocId(key)); 
			if (null != imageDoc) {
				if (thumbNail) {
					return getBytes(imageDoc.content().getString(FN_THUMBNAIL_FILE));
				} else {
					return getBytes(imageDoc.content().getString(FN_FILE));
				}
			}
		}
		return null;
	}

The doc id is: Photo:Image:6356CFB0054B2599C1258473003D9AC3 - and it exists. If I try to open it in the db-admin app it returns the object (quickly):

META:

{
  "meta": {
    "id": "Photo:Image:6356CFB0054B2599C1258473003D9AC3",
    "rev": "62-15c3b7c9dfab00000000000002000000",
    "expiration": 0,
    "flags": 33554432,
    "type": "json"
  },
  "xattrs": {}
}

And the DOCUMENT:

{
  "file": "/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAMgAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5e:
   :
   :
  /0u5SZZfliO0L9KoQRfaJZGkAPzdT1raEbK7ZqrROjjmWX94km7ecDPb2p0kkkU8+bcSqBgZPAPrTbdoV04QtHgbt+4dQAKfCpuoHESl9y/MN21mHoPfvWUGrtl0nrczAph3zXK53/NtwDk+3pxVuTUraJEtHDhW+bCtyBVCaxmtLtIo0lWPIYmXhsfSmQwx3F7LdXGVggOcj+LH8NbtKTuVKXvWOlsIktra6fYjJKFlQE4fCnP4j/Gqd5NJNOpeMNEBknHCL/U1Vs7+W8unnkhPlorKAv8CY/lTXvd+nuFQSAnBBJ6VlZqV2VzWevUtx3VvvimkdkiJONv3lHb+ddDpcDu8N8siSLHKWVt3KDPf1Brj7WRroJGsUWyHORg4Of/1Vswvd2ulSy24j3QnEiqMZHbpQ42ZdNnb64sbQNPtEgfPlfNtAJ964uFUExhRSTgO3lt8zDqSB1z1roRdm90WxlaPjh9rHjO04z7Vzsq3C3L3M+4XnmZ3BvljweCD+HShGzNPS3aO3u7e2v4pp45xJiVSOoxtOePTHNOFu0VrdMYYxLKzcD7oQ49PxqEIRpd/eRgNLPEpcIMZfKncPaoJ3lt3gkj3o91bCUkHgeo/PJqrdSRlzFbhIpYXktZsDlDvjkP4c/pWprNxJJpduXCSzoNsgAxuyOev+eKhtkvmDRC6BjdS0VxHGAR6jjoao/wBrS6e86+W88EeBGsq7SWXhsdz1JzSvcNidLKG105oGlys4BSBV+ZQere2P1rP+2X2lXCvdMJLdcxlgBh16fN61ZgvHgvFV1aZJwHKyEh0J6ZxwR0q1dxmFldgPJf5JIm7e4469KL2eo2rjHtEkiW7tTsj2gjIyVHcE+n0qPTJ43huo85kZl744JqK4+0WFubcs5g5EZPKkHrg5yPpWlb2ZhgidQsiOmJGUf6s/1Ge9S0Z3vIqsUs76KCE742yCAOmPrUS3MkreZDaCRtu4RlSzdfXtn61PKXjubWNgAI2/eSMemf7o96jW7uRcNI9xdyQKB8u4qGHpxQkaNk90klpavHxBczFWOEGV+Vj+fGaztMF9BfvhJ5rYR5kkmfcHB+vAqeK7jkklWcOhDsVVVzyRjGPxpt/NcQ20en2bpCxYPM2M5Y/wkjsOn5079CUWdKWaHUTGYWQHKkxkMrdwc54/GtHVo1eOKVlBXcFZWHXnj6Vh3At5PKE4jLlR80C8nHXr2rqyQ9i/lFFcqQpwGAPbg/hWM3ZqRMtGmccLRl+0TRSfxGRM8FT3B7VBDdtqN1JDtMdyDujZeF3d+Kt3Jm+yyXK6cpmjP7wjcu7P8QGcVJpUMVxKJAJFmeMldnByOfz/AJ4rVy6julIjW2VXxHLJaSEEMq5AJPXHbr2ouJb+K3SK1kZkVh5zEAbgMdB2/matwNbXOpR/b5o3WVwPmyCT09MfrV+6nhM9xBIiAKdqSBgQRno3v71nNtFyjdXOVvZ7m5Kwhmdd2ME9/WnraR2lhI9zF5oLDLIPmTGec9hVdbp7XVmIIJVypRhkMemCK1JmEuAjB5F/1YHBAHBA/PoauC5VY56cE22YL2Uts5uIZRNbkZMiLuA+o7GnyTW9+EZS2JG2yHZyMdH/AMa24bKWA3Bil+yyvGWKfwn6j8azsLaRPOIllkAKFlUjBPf3/KtOZM05LFq9MNvexQLE5URhFYcbiByfzoqO2itrm2Te3mTKMkk7W3dRg0VnZDszkNrSttRTt7cVet7abe0nkvhBhRt71rIW2/LwB6VIS4APrWkq72SPNc/I5yWxvHcD7PJz/smrL6ZJEkYzHx1Gec+tbglYZILZHekmENxt3xgnu3INH1iXVAqpTskht4yyrl+gZqtwu24kk0fZIQ37t2T2PIpr28sfzMDt/vAZFZOakyW7sfPHb3MZFxHvwMKT1H0NY9/pIiiWWLLAdV6kVoF9527zheRU8MoDjcTJ04PerhKUNgTaMrSIi7yPwABjNdDp2yQtn7qkAVXvYY7SB5IItgY72APAqxpy+Rp3mN1Y5qKsua8gldss3Fz5s62yDpycUniaxSTTIdQWQ7hlCuOKh06Mokl9J96VtsYPt3rWli+1+HJ4cAlcsPqKwvySVhxVjI0MkwLJIOVHfsKr6pe/aWZRnA4GakJ+z2aICAW6/Ss4IZ70ID8pNaRV5OQoomv5vLtbWAD7sW81Bp77LmydukgKHPuKfq7r9sO3hVh4FECj7PFICoeMBk3DIzW0bKJWxNpOy2kubQtkhzgZ6Vp3FrGFjgRF+U72H+0aitbGOO/uLl1AjHznPc9qpy3TXDsySld5OWWs5e+9BGi+q2OnSpHK5ZmOGKc7ff8A+tWfewTCUXFxN5m5iYJE+6y+3pWfKbeNtkSKX55PJzVbTtTuhI9kX3W8hyUYZwR3Hoa1jRsro2SSiaIz9pEgSIuBnceG/OpLqaW4i2yxKPQjsaiks5FUXGCY3O35W6f4VGXlLsqsWQDjP8NTZPUhX3IUM1k+YyMZDMWFX5NOe+UXNl5ReRd2xm25z6VXcS3FumFbaG2kY/WlbUBa3cUKJviQ7GGOg7/rV6vVbmkX3KkNxe6NfhmiKP0eNujCtLV7LfGt/aLmFxlgP4DUmranH9lFvcLuxgp8u449VbqKu6LNZ/ZmgtrnzoWORHIMOueoNKUnbnsKcUtYnKFSysCc+ntUjCWewcZJ8sD5farF/Zy22qvaIjMrndGcdR71Kwj0+MgyZkcYOD2rTm2sSnYw1bZID+BFauroF0/TzngITj6moGijbDsoOfu1palZz3jWFvEm7MPUDgVUpLmTYXKNq014I/KUtIPkwvOa0pfCd+LTz3eNCOif/XrdsLbTvDVmpklXz5B8xJ5b2ArK1LxjOx220KiLOD5nJP8AhWHPOUv3a0KMTU4praK1eRCsgUofwNXdOuGLRPGOGOGB9O9W/tltrOnsSgDocsp7fSs6z3W12seQwJ4NNybjZrVGcnfQ0dZYvaRIB1kBFN1eJhauqyFcP8y+vtVpIWu76ziI3DzMkewqPVduyUses7KPrkiog9kJbGBbRrHul7jgf1rd8N6jK9tPa3TjyGYmIv2Yc/lWFMGEaQp95jtrTtrYEpGp+UDav+NbT1i7lN6FyPynu5REpWJ84B7VXu5AjAHptrT3QvFHbwRBGhyWbu5PrWXqpVLNLkYYINp+tcsfekgjHczraQQpLEUUlvnLZyQajsiZZGVByvaptNs5LtTM0bIrfec/xfStK0ls7C63OhMZz8y9M+/rXTOSV11K30HiaC2sy8ieZKQVSLoCfesSO7ns9WZZgcS7RIpGOorqrayt9UtWV5DH5zNiQICVH0rHvdH1HTB88cepWKcgjJKD+a/yqaEoaxe5UVoacU11Kws5GEtuPmBPIAHb/wDVV210ex1yCWCCdbeWFGdFcB43wM891NRwQRWWmm4G+NrnlY3OdgPUA/rUekT2kXiKESYRm+Rjn7wPHT8aiL952NaesrMziosNMvtyhZpytqoU5Gc5c/TAH51hJI8MjRbcEsRz7V0viDTZLW/t7OORWkt/3jr/AHixySPwxVDxBYRx6jDHF8u+BZMjkknrW8bP5lThcoR6kYCSVxj0FatvqysQq7jvHIAHP1qa50KFPCVzdRjfLHMjebtyTGVPBH1rBs5E8kYRVQq3zMeSQKHTjJXI5dbM7db1o9PiuXffCQSoU5xjg+3en7LW7jik3lVBDBVI6jpkVjJLAmg2EJbaSzbgepz/APrrMstR8i8ImIMLjy2JGMY7isFS6o2sl1Og1C2nj8P3qFgrSFETB+U5Ynr+VVLu8kh8MWSSSP5sW6NgoyBznn1pY5ru2inhkIc+apQtypXHb2qW4uLVtMksWTc6MpOBkJnOec81abVkxt9yv4f1KGd5Yf3cbOM5VSoIHXjOAatXZlt9RjuJ40eObhJC5Pyn+7WL/Z0lm4kg2OC4IaHnj39KvNeT21n/AGfcKZIlYSxKg5C5zyx4AzVNK90Ck9mWpb2G5cAARzH5VjYEEgcdcnJ49BV+VbhjFudcKwDIWySMDr34xWHPM8Mkjxr5YcZzFxuBGfvnr9BSZkg02YJlZ9geQ7uRntSaG59C3f75L8bWZkYArt9AOv1xV6yvfsarcQsZLaXiRG4w3r7H+dc2bt0tojvZSigZJyR9Ku2l+Zw0UdthN+2WMnh1Pr78UknuZwerZ02pRi5QT2aI0qYLKOAVx1+o9KxrpbiaBEtuXcebkcKDu7frWhYNcQu8tvGj2hY4RgSyD0z3FWbny5bdLkTb4UfcOBuUD+E+vPeoUkzVNMzr/TrhI2uSAyeZvbnk9BkfrULwyxoJI5yRKRyOdp+lPvvJkCw4LMclVXoD36enSq6pcJbxyyPHHsONp43L6YpN2FJpEskIukVIp8yROdsiqDu45H6VsaNdm4tzC6bSgyPpWZFFYm3kDu8BcZ3KMjjnj1qxpt3FO8MkTNnkFmGN6j1HrWc/fiyeZSRpa2Ve2tpEG2WMmNhvK59+P61z9lKPMeNd8UgJI56fStTVzK1uHiZtysCQgBJ7Vz+pXLW14krSo0i/exkkr9Ogop+/C3UG04mnZ28NzLBbSHe8dwHBA6etZ+orMbi5kVCrL7580Z4PHpW5pWx2k1BPuwwtIcD73HH41mCSJzb3SMwRgc5PT2PtVxlc0TUonO3RdtQ5G12AJPoSO9bNlqNxGht5TGgXgMYwc/n9KivrVGnMkkZTecrIOhJ7Z9Km+2taRrDtjlDAgFgG57f1rR6rQlRSLElzL527FvsMYIZMg4Pse9Q3WYwCMJu/jVQT+NVrmKa4jjvLRiPMjAZUHv0IpkhljRTLuDA5B9DSsir2RbuGtoYRcuRG5XCqvQn3orHWJrmb96Sct0xwKKaijJzk9T//2Q==",
  "datetaken": "2019-09-12T13:12:53+0200",
  "ispublic": true,
  "thumbnailfile": "/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAB4AFoDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD2JCBUu5iuO1PaNAePxqZI02802wKDMTwBTU61caJecUzyhnNIDP1GMwrFqCnBtyTJ7xH7/wCWA3/AauC2DN/WrGFKlSAQeCDVLS38lJLFzl7UhFJ6tGfuH8vlz6qaALaRMp4OBT2O3ApfMDGhgGxz0oAgMp5xT0+dfm5pREuCTSZC9DTAVIkp3kpUfm88Gl8wetAETRyB+5zVgFgmCvNVbW6KDD8mrD3Ks3DACgCJnOehqG4vI7SB57qVIYkGWeRgoA9yatFsDcrLXE/FFt/w/wBSzgkGMjH/AF0WpbsrgW5fiL4XhYqdVifAJYxqzAAd8gYqNfFWk32o295p2pwSjHlTRjKsUY8Ng4Pytj6Bmr5stLlvtBTsY3UDuTtOB+eK1PDvl3WpQwOm6N3C4JPr/wDWx+NZuckUkmfVCk980jSfNgE/WsXTb66aBrecM09s3lSMerYAKt+KkH6k+laiSo+BjBPWtSSw03y4FQ+YamjgEp4NK1rjjP5UwK7PUBdsnn9avCybGcce5p32DPYfnQBzsn2mEcnd9DUf2uZVyyHH1rehmt5eGyB61HPHZOrAnj3GadxGSl+SMHI/Guc+IFwJvBGoIDz+7P8A5EWukktbNTlH/wDHa5zxtDF/whupFZBkKpAx6OtTP4WNbnz5Gm28iI55Vh+de1+EdE0JvDMWqLbCO/MWJfmJO/f1AOQM4HQdDXlmhaaNS8QadZDhZdnmH/Z+8x/IGvQL/R9XmurhtH1KC30yeVpFIU71x8rKOwGe4rGHvM0jazO70jXxqtwurpC0FvJP9hlDHnsY2OenzMy/8DHpXXLlBnj61wXhKwT/AIQ86fM2FkeUOVPIJY8g+o4INbmlambqwVpW/wBJiYwzL0G9TgkD0PUexFbWMzoRM+cZP4VNHKY23EZrHW8c8lDj/eFPN42Put+ANOwGtPPuTOSvHrWebls/6w1WMzOuCGwaj496LAefQfEooP8ASdIkjLEbQC2QPf5asR/E2zePK2+X93Kf+y1yjatfXpcJJhnHOAPm9f61WELzZLs0rE9QcZ56CvPliZdGF1bY7qLx+JcM1iCh6FbjP81rP8U+Lo9Q8PXdnFabfMTDM0mSOR0GK5MSwgkOclfX/PFNu5ybKXaPlKNnOOPp7VDxM2rNkqWpW8CW4uvEWN5VhZPhh/Dn5M/+PGvTrkDT9ItbPIwHC9Pqa85+HG1tcuGXgraBT7Zcf4V2viK5KLE2eko7+vFdtDY1IbDxPb2CXFlPFOwSdstGfp702Lxfp1jrLSpFci3u0AkVwDiRfukZPdcg/wC6tcgkrNfanIr7sXRwB2yo/T/Cmzus9u8TMVc8qyn7rDofwNc88TOM3HoQ0d/L49RVJgtGYg8bn28fkarHx/OV508def8ASf8A7GuIg1eF0jVztkIwUPO05wR+fH5U1NRjeWZZGG5GwowM4IH9c1LxNVBZHcp4+m53aeAM8FbgHI/75q6PG9nj/V3P/jv/AMVXAebbOAUZOnQrTMWvcx5+gpfW6gGlawlJQkSrvIJ+b13c/ToeKsLdLbw2iOMsE2tgcLx1/wA+tYtrrLiBOqsPmDk5zz3/AO+sj6Vn32oXE0/7tgSo3OdwAVfcfU9PpWHKwbSWh0s32S5szKqLkZxxnknnHbAIPH+FYuosLnQrhYXMckSkgkfe9v0/nWSl1cREwBiMEggHG1iDnp9T+VPE6/ZCu7/WAquT1GfT8KtQcXchv3kaHw1E9rqN68gXmNVGBjvk/wBPzre8WX3+jlScYZT9Oaw/B88g1i6tSQENsZAMc53KKPFE/D8kHIwfSu6nJvU2eiMrSLx2vr6YxM8LsXLEcDBwPx/wrUE6kqpjU5AH3fmB/wAeaXTlSPw/FG4RCwWTOB/EVJ7e+T7VJJYKFygBO/dsyOVPUfXpXFValJsynF7mRcmGDXBhiu9FI7/NuXPP0H6VMIU3yyqA534B6ZGOar6lBm+ukVy7pAjoWXkEEkE0tm7SJJKchSzbo8dDn/H+VD+FMd3Y0PPgjRT93jtzj0z7UwquTnbnvhTUDKm0qVIUg9OQDUBkRCVK5I4JyKhJMTHTbztUW5C4LEDsOPT1yBUckexWcqMOdzd+OMAfj705bwyxsSo3nk9uOvP86ZOZLu22xsq4O3bgjP4gf5zVq+iB2tYmBChAE+dzuDAYBBByD69ajOxplXcFcRgNk9fpjj3qNpneLy4wAA5yW6jHtUX2aKa6E5uH3nIUcYOBwMD2oS3uSr3VzW8KSxp4keV5Ni+QY/qSRgfp+lJ4tysjRjglsCuVmupopm+zyFQ4BJAGeK6XWZDcXdn5yneVVpEY4OcAkGuyPuxNpO5oxXdsUt1b5SqbSMcdMflwKpy3u2SZXZldWJBxwcA4H9PxqnPcwKJISGVwSSrttAzzx/8AXNZ/2pmlNvKzKxG1MDJIx39v5fSuNU3LUzlJs27q6huvOu4CyuYhHIpH3cBuh6Ec8+lMtCj3DpjidnkTd6hiGUfUbT+dZ9u8olIZZAceW+4dj0J+lSy2s32SEZMU8ZLA9QTyRz6EA05R6De1zdgVYmlL5AKfOB1UZHOe46dP6USWumPIzGKcEkkhWUAfSqU988EzxFTICuBIM5wVzxz3wPyp665booU2iOQMFieT71moytoQ9NGZaylFaGVS7MAxIUDA9u4/+vVvzyFKxNGgC42gdOe3NZKXuLfyJZECK53Pu5/OiScTB44QyptydoyT04yOvetuRju0Rfa7i3uyxkOxyVGDnPTv+NaUU0b227AV1Vm/UAGoIbOKNkijKFlyZEl6gcDpyP8A9dTG38uC4EUZwVGz5gQVDZOCPbtRNxeiF1RjuI3vtzj5VKknPbmtWS+iutQNxNccghiR14rNaJZmlCMpAjDFj261etUW2ki3IrjBDAj7w/z/ADrdzSib+ZDqcbQv9piO+1dd0e/5uOQRnqMYx+NU0uo8LFBblWPV8bifYDOAPzNdRY28BhS1lB2umNqnIz9KrXOhQgMY9yru+XZ6/SpjOJSpX1RhxQyR3KvJOSU+7lwCPY56Dn37118SW+o6HIltbSJdwfNIFywZcMScenbPb05rlLnR7q2y0apMoyGVUGcfh+PSruiahdWM0MyTzW5hDqoBxtDAgnB6joceoq2otakuD2JvKmtb9Le6dWZ1KwiNg3kkclWx35Bx2rREQcbvLXnnov8AhWZPbLcpDJHdRvIkoferZbPrg8/X8K3206SRi4jDhjkOGGG9+tctW2nQSpuSucI8F1dy5MQiQ889APStfRoHgtZpHXEi8Jn+HkDd+uaKKupNuNjncr6EUlpE8rTxSlAPlZ5T15649cflinzsCoW3eXdH8ysT944yeO360UULXcpK5BEkLwz4gmVT94gggfQEf1q2QpvA4PygbzGRkY9QfqOlFFOQ22SW1xO6L5aMSxzySCVB7H8elaNnqEU0Lb8rIrgAMMZHP68fpRRRbQ6aUmkhfPtyyNuUEYGSAMnPWnm2SafyXJ+YHg9Ce2fUGiipm7LQ1uZM+nW6yxyp+7dlJ4HyDHb+f5VP9tk/5+APYNgCiij4lqHw6o//2Q==",
  "type": "Image",
  "parenttype": "Photo",
  "key": "6356CFB0054B2599C1258473003D9AC3",
  "userkey": "2124DEFEC111BA8FC1257ED20034B387"
}

So is this all driven by timeouts? If so, how do I check and adjust accordingly? Is this related to the latest version 2.7.9 of the Java SDK?

Or is it about corrupt data? If so, how do I verify and correct accordingly? … and how could it start at all? Is it the Sync.Gateway that has done something “forbidden”?

I would really appreciate some thoughts and ideas here :pray:

FYI I moved this to the proper topic (Java SDK vs Couchbase Server). Hopefully that will get some visibility.

1 Like

Ok, thanks… I think my perception of the problem has evolved a little since I created it…

Thanks for adjusting the category - I’ll wait in anticipation!

Ok, I tried to downgrade the SDK on the server to 2.7.4 - restarted the server (but did nothing more) - and it still fails… (with the timeout error).

I have kept this situation in my environment in the hope someone would have any suggestions… but if noone has a suggestion (which I find really worrying) then there is no point in that - and then I suppose I’ll just have to find a backup and restore… But then the source of the problem has not been found - and I already had this once in the same environment back in May…

I saw this on Friday after the move, but did not have a chance to reply.

If the failure is in snappy decoding, that likely means that for some reason the framing indicates that it is snappy encoded, but it is not. We’d probably want to verify this.

There are a couple of ways of going about this. One is you can install libcouchbase, and you’ll have the cbc tools. I think cbc-stats will do it.

Simpler might be to just try to get() it with a super simple test program. Just bootstrap and get it with 2.7.9. Turn the log level up to debug, and we will see the specifics of the framing. We’re looking for the datatype byte, which will indicate if it’s compressed.

Checking with someone else, cbc-cat is probably the quickest one to use out of the box. See the docs/man page.

Thanks for responding @ingenthr!

Just to be sure, should I install the libcouchbase on one of the database servers - or should it be on a different server (e.g. the SG server or my App server - the one that uses the Java SDK)?

Ok, I installed it on one of the database servers. This is the response:

Photo:Image:6356CFB0054B2599C1258473003D9AC3 CAS=0x15c3b7c9dfab0000, Flags=0x2000000, Size=111779, Datatype=0x01(JSON)
{"file":"/9j/4AAQSkZJRgABAgAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAMgAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWW
: 
: 
60UULXcpK5BEkLwz4gmVT94gggfQEf1q2QpvA4PygbzGRkY9QfqOlFFOQ22SW1xO6L5aMSxzySCVB7H8elaNnqEU0Lb8rIrgAMMZHP68fpRRRbQ6aUmkhfPtyyNuUEYGSAMnPWnm2SafyXJ+YHg9Ce2fUGiipm7LQ1uZM+nW6yxyp+7dlJ4HyDHb+f5VP9tk/5+APYNgCiij4lqHw6o//2Q==","type":"Image","parenttype":"Photo","key":"6356CFB0054B2599C1258473003D9AC3","userkey":"2124DEFEC111BA8FC1257ED20034B387"}

Does this give you a hint?

I did some more testing to try and see if I could produce a consistent case.

So first I added some more logging to my code so I could see exactly what id failed. Unfortunately, it is not consistent…

One thing I noticed is that the coded refetched the same record several times instead of caching the record in memory. So for some simple records (articles) used to build the frontpage it read the same document from the database more than 5 times (due to referencing various properties of the same record in JSP). So I changed the code to cache the articles - but it made no difference.

This is the stacktrace when I got the error reading Article:5:

Script interpreter error, line=1, col=9: Error calling method 'getSidebarArticle(java.lang.String)' on java class 'dk.dtu.aqua.catchlog.bean.ArticleBean'
->   1: Article.getSidebarArticle(compositeData.articleId).getSidebarImageClass()
com.ibm.xsp.exception.EvaluationExceptionEx: Error while executing JavaScript computed expression
	at com.ibm.xsp.binding.javascript.JavaScriptValueBinding.getValue(JavaScriptValueBinding.java:132)
	at com.ibm.xsp.page.compiled.ExpressionEvaluatorImpl.getBindingValue(ExpressionEvaluatorImpl.java:151)
	at xsp.SideBarBox$SideBarBoxPage.createImage5(SideBarBox.java:223)
	at xsp.SideBarBox$SideBarBoxPage.createComponent(SideBarBox.java:114)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.buildComponent(AbstractCompiledPage.java:265)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:115)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIDataPanelBase.buildContents(UIDataPanelBase.java:459)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIDataPanelBase.buildContents(UIDataPanelBase.java:459)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIIncludeComposite.buildPageContents(UIIncludeComposite.java:692)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:340)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.createTree(AbstractCompiledPage.java:256)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.addComponent(AbstractCompiledPage.java:389)
	at com.ibm.xsp.component.UIIncludeComposite.buildContents(UIIncludeComposite.java:453)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildFacet(CompiledComponentBuilder.java:205)
	at com.ibm.xsp.component.UICallback.buildContents(UICallback.java:180)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIDataPanelBase.buildContents(UIDataPanelBase.java:459)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIIncludeComposite.buildPageContents(UIIncludeComposite.java:692)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:340)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.createTree(AbstractCompiledPage.java:256)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.addComponent(AbstractCompiledPage.java:389)
	at com.ibm.xsp.component.UIIncludeComposite.buildContents(UIIncludeComposite.java:453)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildFacet(CompiledComponentBuilder.java:205)
	at com.ibm.xsp.component.UICallback.buildContents(UICallback.java:180)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:345)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:345)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildFacet(CompiledComponentBuilder.java:205)
	at com.ibm.xsp.component.UICallback.buildContents(UICallback.java:180)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:345)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:345)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIIncludeComposite.buildPageContents(UIIncludeComposite.java:692)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:340)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.createTree(AbstractCompiledPage.java:256)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.addComponent(AbstractCompiledPage.java:389)
	at com.ibm.xsp.component.UIIncludeComposite.buildContents(UIIncludeComposite.java:453)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIIncludeComposite.buildPageContents(UIIncludeComposite.java:692)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:340)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.createTree(AbstractCompiledPage.java:256)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.addComponent(AbstractCompiledPage.java:389)
	at com.ibm.xsp.component.UIIncludeComposite.buildContents(UIIncludeComposite.java:453)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildChildren(CompiledComponentBuilder.java:123)
	at com.ibm.xsp.page.compiled.CompiledComponentBuilder.buildAll(CompiledComponentBuilder.java:84)
	at com.ibm.xsp.component.UIViewRootEx.buildContents(UIViewRootEx.java:1649)
	at com.ibm.xsp.component.UIViewRootEx2.buildContents(UIViewRootEx2.java:247)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.initComponent(AbstractCompiledPage.java:334)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.createTree(AbstractCompiledPage.java:256)
	at com.ibm.xsp.page.compiled.AbstractCompiledPage.createViewRoot(AbstractCompiledPage.java:167)
	at com.ibm.xsp.application.ViewHandlerExImpl._createViewRoot(ViewHandlerExImpl.java:521)
	at com.ibm.xsp.application.ViewHandlerExImpl.createViewRoot(ViewHandlerExImpl.java:567)
	at com.ibm.xsp.application.ViewHandlerExImpl.doCreateView(ViewHandlerExImpl.java:142)
	at com.ibm.xsp.application.ViewHandlerEx.createView(ViewHandlerEx.java:90)
	at dk.dtu.aqua.catchlog.control.AppViewHandler.createView(AppViewHandler.java:167)
	at com.ibm.xsp.webapp.FacesServlet.serviceView(FacesServlet.java:250)
	at com.ibm.xsp.webapp.FacesServletEx.serviceView(FacesServletEx.java:157)
	at com.ibm.xsp.webapp.FacesServlet.service(FacesServlet.java:159)
	at com.ibm.xsp.webapp.FacesServletEx.service(FacesServletEx.java:138)
	at com.ibm.xsp.webapp.DesignerFacesServlet.service(DesignerFacesServlet.java:103)
	at com.ibm.designer.runtime.domino.adapter.ComponentModule.invokeServlet(ComponentModule.java:588)
	at com.ibm.domino.xsp.module.nsf.NSFComponentModule.invokeServlet(NSFComponentModule.java:1335)
	at com.ibm.designer.runtime.domino.adapter.ComponentModule$AdapterInvoker.invokeServlet(ComponentModule.java:865)
	at com.ibm.designer.runtime.domino.adapter.ComponentModule$ServletInvoker.doService(ComponentModule.java:808)
	at com.ibm.designer.runtime.domino.adapter.ComponentModule.doService(ComponentModule.java:577)
	at com.ibm.domino.xsp.module.nsf.NSFComponentModule.doService(NSFComponentModule.java:1319)
	at com.ibm.domino.xsp.module.nsf.NSFService.doServiceInternal(NSFService.java:662)
	at com.ibm.domino.xsp.module.nsf.NSFService.doService(NSFService.java:482)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.doService(LCDEnvironment.java:357)
	at com.ibm.designer.runtime.domino.adapter.LCDEnvironment.service(LCDEnvironment.java:313)
	at com.ibm.domino.xsp.bridge.http.engine.XspCmdManager.service(XspCmdManager.java:272)
Caused by: com.ibm.jscript.InterpretException: Script interpreter error, line=1, col=9: Error calling method 'getSidebarArticle(java.lang.String)' on java class 'dk.dtu.aqua.catchlog.bean.ArticleBean'
	at com.ibm.jscript.types.JavaAccessObject.call(JavaAccessObject.java:335)
	at com.ibm.jscript.types.FBSObject.call(FBSObject.java:161)
	at com.ibm.jscript.ASTTree.ASTCall.interpret(ASTCall.java:197)
	at com.ibm.jscript.ASTTree.ASTMember.interpret(ASTMember.java:106)
	at com.ibm.jscript.ASTTree.ASTCall.interpret(ASTCall.java:88)
	at com.ibm.jscript.ASTTree.ASTProgram.interpret(ASTProgram.java:119)
	at com.ibm.jscript.ASTTree.ASTProgram.interpretEx(ASTProgram.java:139)
	at com.ibm.jscript.JSExpression._interpretExpression(JSExpression.java:435)
	at com.ibm.jscript.JSExpression.access$1(JSExpression.java:424)
	at com.ibm.jscript.JSExpression$2.run(JSExpression.java:414)
	at java.security.AccessController.doPrivileged(AccessController.java:730)
	at com.ibm.jscript.JSExpression.interpretExpression(JSExpression.java:410)
	at com.ibm.jscript.JSExpression.evaluateValue(JSExpression.java:251)
	at com.ibm.jscript.JSExpression.evaluateValue(JSExpression.java:234)
	at com.ibm.xsp.javascript.JavaScriptInterpreter.interpret(JavaScriptInterpreter.java:222)
	at com.ibm.xsp.javascript.JavaScriptInterpreter.interpret(JavaScriptInterpreter.java:194)
	at com.ibm.xsp.binding.javascript.JavaScriptValueBinding.getValue(JavaScriptValueBinding.java:78)
	... 97 more
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: {"b":"data","r":"192.168.42.211:11210","s":"kv","c":"78B1DDA0CAEBCCCD/000000003DD8581E","t":2500000,"i":"0x42a2","l":"192.168.42.226:56714"}
	at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
	at rx.observables.BlockingObservable.singleOrDefault(BlockingObservable.java:372)
	at com.couchbase.client.java.CouchbaseBucket.get(CouchbaseBucket.java:139)
	at com.couchbase.client.java.CouchbaseBucket.get(CouchbaseBucket.java:134)
	at dk.dtu.aqua.catchlog.dao.CouchbaseArticleDAO.loadArticle(CouchbaseArticleDAO.java:96)
	at dk.dtu.aqua.catchlog.dao.facade.ArticleCRUDFacade.findArticle(ArticleCRUDFacade.java:74)
	at dk.dtu.aqua.catchlog.bean.ArticleBean.getSidebarArticle(ArticleBean.java:272)
	at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at com.ibm.jscript.types.JavaAccessObject.call(JavaAccessObject.java:322)
	... 113 more
Caused by: java.util.concurrent.TimeoutException: {"b":"data","r":"192.168.42.211:11210","s":"kv","c":"78B1DDA0CAEBCCCD/000000003DD8581E","t":2500000,"i":"0x42a2","l":"192.168.42.226:56714"}
	at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:131)
	at com.couchbase.client.java.bucket.api.Utils$1.call(Utils.java:127)
	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
	at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166)
	at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191)
	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
	at java.util.concurrent.FutureTask.run(FutureTask.java:277)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:191)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(Thread.java:811)

The output of cbc cat Article:5... is:

Article:5            CAS=0x159cf2e1e14f0000, Flags=0x2000000, Size=1122, Datatype=0x01(JSON)
{"articletype":"1","displaytype":"3","heading":"Vi sladrer ikke","inactive":true,"ispublic":true,"key":"5","link":"/pageInfo.xsp","quote":false,"revisioninfo":{"created":"2015-12-04T10:07:53+0100","createdby":"John Dalsgaard/BA171123846CEBF1C1257CB2002DA330/Fangst","modifiedcount":6,"updates":[{"modified":"2015-12-04T11:40:49+0100","modifiedby":"John Dalsgaard/BA171123846CEBF1C1257CB2002DA330/Fangst"},{"modified":"2015-12-08T11:58:33+0100","modifiedby":"John Dalsgaard/82DCD38079E76A01C1257CB2002E2014/Fangst"},{"modified":"2015-12-08T13:45:10+0100","modifiedby":"John Dalsgaard/2124DEFEC111BA8FC1257ED20034B387/Fangst"},{"modified":"2015-12-11T16:09:42+0100","modifiedby":"John Dalsgaard/2124DEFEC111BA8FC1257ED20034B387/Fangst"},{"modified":"2019-04-27T12:26:08+0200","modifiedby":"John Dalsgaard/2124DEFEC111BA8FC1257ED20034B387/Fangst"}]},"showarrow":true,"sidebarimagename":"lockwhite.png","text":"\u003cp\u003eNogle ønsker at holde visse fangster for sig selv mens andre gerne vil vise sig selv og fisken frem.\u003c/p\u003e\u003cp\u003eBegge dele er muligt på Fangstjournalen.\u003c/p\u003e","type":"Article"}

But sometimes Article:5 succeeds - and it fails on one of the other articles - or the image used by one of the articles.

I tried to investigate the “timeout”…

I’ve set the addresses of the database servers directly in the hosts file of the app. server:

cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.42.226	fangst.dalsgaard-data.dk
192.168.42.211	db1.dalsgaard-data.dk db1
192.168.42.212	db2.dalsgaard-data.dk db2
192.168.42.213	sg1.dalsgaard-data.dk sg1

And a ping responds Ok:

[jda@ds3 ~]$ ping db1
PING db1.dalsgaard-data.dk (192.168.42.211) 56(84) bytes of data.
64 bytes from db1.dalsgaard-data.dk (192.168.42.211): icmp_seq=1 ttl=64 time=0.183 ms
64 bytes from db1.dalsgaard-data.dk (192.168.42.211): icmp_seq=2 ttl=64 time=0.178 ms
64 bytes from db1.dalsgaard-data.dk (192.168.42.211): icmp_seq=3 ttl=64 time=0.219 ms
[jda@ds3 ~]$ ping db2
PING db2.dalsgaard-data.dk (192.168.42.212) 56(84) bytes of data.
64 bytes from db2.dalsgaard-data.dk (192.168.42.212): icmp_seq=1 ttl=64 time=0.271 ms
64 bytes from db2.dalsgaard-data.dk (192.168.42.212): icmp_seq=2 ttl=64 time=0.182 ms
64 bytes from db2.dalsgaard-data.dk (192.168.42.212): icmp_seq=3 ttl=64 time=0.568 ms

So not sure that the network setup is the direct cause of the timeouts…

Ok @ingenthr , I now have a consistent case :slight_smile:

Situation: I have an email and I want to find the key for the user and then load the entire user. The user info is split into two documents:

  1. User that contains the “non-personal” info of the user
  2. Private that contains personal info. This makes it easier to control where the personal information goes (e.g. in the Sync.Gateway).

So first I search for any Private documents with the email address. If found then we have a key. We then use that key to lookup the User directly by using ...get("User:2124DEFEC111BA8FC1257ED20034B387") and if found we get thePrivatedocument directly by using…get(“User:Private:2124DEFEC111BA8FC1257ED20034B387”) - and we then combine the two documents into one User object in the code.

So with the above in mind this output should be clearer:

CouchbaseUserDAO.loadUser: Find by email: john@dalsgaard-data.dk
BaseCouchbaseDAO.getAllFromDb: Query=SELECT * FROM data WHERE type='Private' AND (lower(email)=lower('john@dalsgaard-data.dk'))
BaseCouchbaseDAO.getAllFromDb: Result from Query=N1qlQueryResult{status='success', finalSuccess=true, parseSuccess=true, allRows=[{"data":{"address":"Solbjergvej 42\r\nSolbjerg","cellphone":"40566308","city":"Høng","country":"DK","email":"john@dalsgaard-d
ta.dk","key":"2124DEFEC111BA8FC1257ED20034B387","name":"John Dalsgaard","parenttype":"User","password":"x5w77zZoW3","phone":"49141248","sex":"M","type":"Private","userkey":"2124DEFEC111BA8FC1257ED20034B387","zip":"4270"}}], signature={"*":"*"}, info=N1qlM
trics{resultCount=1, errorCount=0, warningCount=0, mutationCount=0, sortCount=0, resultSize=348, elapsedTime='6.388261ms', executionTime='6.284091ms'}, profileInfo={}, errors=[], requestId='016d8d47-7303-403a-8774-687df685a030', clientContextId='dcda9323-
4f2-4f0b-9176-4a1efe13cc4c'}
CouchbaseUserDAO.getKeyByEmail: Result={"data":{"zip":"4270","country":"DK","address":"Solbjergvej 42\r\nSolbjerg","city":"Høng","sex":"M","type":"Private","userkey":"2124DEFEC111BA8FC1257ED20034B387","password":"x5w77zZoW3","phone":"49141248","name":"Joh
 Dalsgaard","cellphone":"40566308","parenttype":"User","email":"john@dalsgaard-data.dk","key":"2124DEFEC111BA8FC1257ED20034B387"}}
CouchbaseUserDAO.loadUser: Found key: 2124DEFEC111BA8FC1257ED20034B387
CouchbaseUserDAO.loadUser: Find by direct key: User:2124DEFEC111BA8FC1257ED20034B387
CouchbaseUserDAO.loadUser: Found user doc. Get private doc: User:Private:2124DEFEC111BA8FC1257ED20034B387
Rest.handle: ERROR - java.lang.RuntimeException: Could not decode snappy-compressed value.
Rest.dumpRawExceptionData: ERROR - Exception: CouchbaseException - java.lang.RuntimeException: Could not decode snappy-compressed value.
   com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:369)
   com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageC...
   com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMes...
   com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessa...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstract...
   com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMes...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstract...
   com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerC...
   com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessa...
   com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDe...
   com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChann...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstract...
   com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler....
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstract...
   com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Default...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstra...
   com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelP...
   com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(Abstract...
   com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:...
   com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEvent...
   com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java...
   com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
   com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEv...
   com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRun...
   java.lang.Thread.run(Thread.java:811)

This is the code that fails:

	Util.info("Found user doc. Get private doc: " + getPrivateDocId(key));
-->	JsonDocument privDoc = getDb().get(getPrivateDocId(key));

And here is the output of the cbc cat command:

cbc cat User:Private:2124DEFEC111BA8FC1257ED20034B387 -U http://db1/data -u xxxxx -P yyyyy
User:Private:2124DEFEC111BA8FC1257ED20034B387 CAS=0x15bd8c734d220000, Flags=0x2000000, Size=339, Datatype=0x01(JSON)
{"address":"Solbjergvej 42\r\nSolbjerg","cellphone":"40566308","city":"Høng","country":"DK","email":"john@dalsgaard-data.dk","key":"2124DEFEC111BA8FC1257ED20034B387","name":"John Dalsgaard","parenttype":"User","password":"x5w77zZoW3","phone":"49141248","sex":"M","type":"Private","userkey":"2124DEFEC111BA8FC1257ED20034B387","zip":"4270"}

I just queried the document from the db admin panel (where the lookup by id works fine). The meta data is:

{
  "meta": {
    "id": "User:Private:2124DEFEC111BA8FC1257ED20034B387",
    "rev": "15-15bd8c734d2200000000000002000000",
    "expiration": 0,
    "flags": 33554432,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "5-a5e4c60689b9662026bb4fb9f2ef1e5d",
      "sequence": 424079,
      "recent_sequences": [
        187871,
        424071,
        424074,
        424076,
        424079
      ],
      "history": {
        "revs": [
          "4-2565a10f9deddeece8201479ce81441a",
          "5-a5e4c60689b9662026bb4fb9f2ef1e5d",
          "1-92b7043cf752430e905306c52b043359",
          "2-7432f4999e86db5c881f69a70036dd7e",
          "3-1b3a53baf64c38677f9effeaebfe94b7"
        ],
        "parents": [
          4,
          0,
          -1,
          2,
          3
        ],
        "channels": [
          [
            "channel.2124DEFEC111BA8FC1257ED20034B387"
          ],
          [
            "channel.2124DEFEC111BA8FC1257ED20034B387"
          ],
          [
            "channel.2124DEFEC111BA8FC1257ED20034B387"
          ],
          [
            "channel.2124DEFEC111BA8FC1257ED20034B387"
          ],
          [
            "channel.2124DEFEC111BA8FC1257ED20034B387"
          ]
        ]
      },
      "channels": {
        "channel.2124DEFEC111BA8FC1257ED20034B387": null
      },
      "access": {
        "2124DEFEC111BA8FC1257ED20034B387": {
          "channel.2124DEFEC111BA8FC1257ED20034B387": 187871
        }
      },
      "cas": "0x0000224d738cbd15",
      "value_crc32c": "0x3adda133",
      "time_saved": "2019-08-23T14:17:52.235653954+02:00"
    }
  }
}

And this is consistent!

Please let me know what I can do to further help?

Thoughts:
This system has been migrated from another database - so in some places I guess it could be optimized. In this case I do the SELECT on the private doc to find it by email and after loading the user doc I load the private doc again. But as I understand it, a direct lookup is quick - so I haven’t bothered to do anything about it here…

This is also a development/test environment - so earlier versions of the sync. formula could also have done “funny” things to the data. I guess you may be able to detect if this is the issue by looking at the meta data. If this is the case then I could just start over from scratch with a fresh copy from the production database (but I want to do this right: Best way to clone a bucket to a demo environment?)

… and this stacktrace is for trying to load an image. Slightly different - but with line numbers for where it broke:

Caused by: com.ibm.jscript.InterpretException: Script interpreter error, line=1, col=9: Error calling method 'getSidebarImageRef(java.lang.String)' on java class 'dk.dtu.aqua.catchlog.bean.ArticleBean'
	at com.ibm.jscript.types.JavaAccessObject.call(JavaAccessObject.java:335)
	at com.ibm.jscript.types.FBSObject.call(FBSObject.java:161)
	at com.ibm.jscript.ASTTree.ASTCall.interpret(ASTCall.java:197)
	at com.ibm.jscript.ASTTree.ASTProgram.interpret(ASTProgram.java:119)
	at com.ibm.jscript.ASTTree.ASTProgram.interpretEx(ASTProgram.java:139)
	at com.ibm.jscript.JSExpression._interpretExpression(JSExpression.java:435)
	at com.ibm.jscript.JSExpression.access$1(JSExpression.java:424)
	at com.ibm.jscript.JSExpression$2.run(JSExpression.java:414)
	at java.security.AccessController.doPrivileged(AccessController.java:730)
	at com.ibm.jscript.JSExpression.interpretExpression(JSExpression.java:410)
	at com.ibm.jscript.JSExpression.evaluateValue(JSExpression.java:251)
	at com.ibm.jscript.JSExpression.evaluateValue(JSExpression.java:234)
	at com.ibm.xsp.javascript.JavaScriptInterpreter.interpret(JavaScriptInterpreter.java:222)
	at com.ibm.xsp.javascript.JavaScriptInterpreter.interpret(JavaScriptInterpreter.java:194)
	at com.ibm.xsp.binding.javascript.JavaScriptValueBinding.getValue(JavaScriptValueBinding.java:78)
	... 62 more
Caused by: com.couchbase.client.core.CouchbaseException: java.lang.RuntimeException: Could not decode snappy-compressed value.
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:369)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:811)
Caused by: java.lang.RuntimeException: Could not decode snappy-compressed value.
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.handleSnappyDecompression(KeyValueHandler.java:397)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:949)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:132)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:338)
	... 33 more
Caused by: com.couchbase.client.deps.org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting at 1
	at com.couchbase.client.deps.org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
	at com.couchbase.client.deps.org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:47)
	at com.couchbase.client.deps.org.iq80.snappy.Snappy.uncompress(Snappy.java:85)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.handleSnappyDecompression(KeyValueHandler.java:395)
	... 36 more

Hey @jda

You can see from the code that Snappy decompression is only attempted if the datatype reported by the server indicates it’s Snappy compressed. But your cbc-cat detective work indicates it’s stored as non-Snappy.

What is your bucket’s Compression Mode setting? I’m wondering if a) it’s Active and the server is snappy-compressing on the fly and b) perhaps something’s going wrong here and that’s why it’s failing.

If it’s Active, could you try changing it to Passive and seeing if the issue is resolved? Passive means that if a document is stored compressed it will be sent compressed, but no active on-the-fly compression will be performed.

Hi @graham.pople

Thanks for responding.

I’ve tried to find the bucket’s compression mode setting - but haven’t been able to find it. The docs I found (SDK) seemed to be related to the Enterprise version. I’m using the Community Edition…

/John

Indeed, compression is available only in Enterprise Edition, where you can set levels. That makes this case all that more confusing. What should happen is the client (whether Java SDK or Sync Gateway) tries to negotiate snappy compression, the cluster will indicate it’s not supported, and the client won’t compress it.

The server also won’t compress it actively and it should not send any responses to the client indicating it is compressed.

So, what’s confusing here is how the client is receiving, occasionally, items that it says are compressed.

I believe you’re adding these images via Sync Gateway, correct? Sync Gateway does have the ability to manipulate metadata like the compressed bit directly. Perhaps there is something happening where that bit is being set and because you’re on community edition and compression cannot be negotiated, it just stores that way passing it on to other clients. @adamf or @borrrden, any thoughts on this possibility?

The only other though I have @jda, if it’s a transient issue, is that you turn the log level up to debug in a test environment where you can catch it. We could try with the log level at debug for the com.couchbase.client.core.endpoint.kv.KeyValueHandler package first.

Ok, interesting @ingenthr

I just tested that I had not by error downloaded and installed the Enterprise Edition of SG. But it seems Ok:

{
    "couchdb": "Welcome",
    "vendor": {
        "name": "Couchbase Sync Gateway",
        "version": "2.6"
    },
    "version": "Couchbase Sync Gateway/2.6.0(127;b4c828d) CE"
}

The way I handle the images (driven by the somewhat inconsistent handling of attachments in the server side SDK and CBL) is that I Base64Encode the image and put it in the document as pure text. And then on retrieval I decode it prior to showing the image. So I’m just saving the images as “normal” documents directly in the database (so not via Sync.Gateway).

I have initially synced the images to CBL - but I have changed that to only load the images on request directly via a call to a service on the server - simply to avoid the large data volume. So you may be able to see “signs” of this in the image document.

… and another thing @ingenthr

The consistent example above does not include an image. So I guess that indicates another problem - and thus the “Snappy” errors could just be symptoms of the real problem??