How to deal with Sync Gateway/Couchbase Server Tap notification delay

Couchbase Server
1

Hi, Couchbase team!

I am making an app based on the Sync Gateway and what I am seeing is that the Sync Gateway notifies Websocket listeners with the 2-second timer. That becomes apparent when you look at the times the Sync Gateway receives notification from Couchbase Server’s Tap Feed:

02:03:35.452672
02:03:41.453017
02:03:45.453248
02:04:01.454168
02:04:05.455399
02:04:11.455745
02:04:19.455205
02:04:27.456665
02:04:33.456011

As you can see, the number of milliseconds is always the same and second is always an odd number, so I gather there is a timer somewhere in the Couchbase Server that’s set to two seconds.

What I’m wondering is how to dial it down to, say, 500ms so that Sync Gateway gets its notifications faster, because it’s really critical for me. Can you help me?

I understand that there may be other implications about lowering the timer and am willing to accept the risk.

1 Like
2

I don’t know of anything off the top of my head that would set a 2-second notification timer, but I’d like to take a closer look. Can you provide some more details on which notification messages you’re seeing in the SG log associated with the timestamps you’ve listed above?

3

I’ve placed those messages myself. To repeat it, just place something like

base.Warn("changeListener: Written event to channel %q", key)

into the src/github.com/couchbaselabs/sync_gateway/db/change_listener.go at the end of an else clause inside the goroutine in the Start func.

4

Or maybe this is too far down the pipeline, try applying this patch to the gomemcached submodule instead:

diff --git a/client/tap_feed.go b/client/tap_feed.go
index faae54d..58c7815 100644
--- a/client/tap_feed.go
+++ b/client/tap_feed.go
@@ -7,6 +7,7 @@ import (
	"io"
	"log"
	"math"
+	"os"
 
	"github.com/couchbase/gomemcached"
 )
@@ -61,6 +62,12 @@ type TapEvent struct {
	Cas        uint64
 }
 
+var logger *log.Logger
+
+func init() {
+	logger = log.New(os.Stderr, "", log.Lmicroseconds)
+}
+
 func makeTapEvent(req gomemcached.MCRequest) *TapEvent {
	event := TapEvent{
		VBucket: req.VBucket,
@@ -289,6 +296,9 @@ loop:
		}
 
		event := makeTapEvent(pkt)
+
+		logger.Printf("** TapFeed received %#v : %q", pkt, pkt.Body)
+
		if event != nil {
			if event.Opcode == tapEndStream {
				break loop

Then try to send one new document to sync gateway and look at the logs, you’ll find that there’s considerable delay between sending the document to the server and getting it back. Then try this a couple times more to see that the server answers at the exact same millisecond number.

I’ve used Couchbase Server v3.0.2 enterprise to reproduce this.

5

Yes, I was able to reproduce similar results in my local environment. I’ll try to get some input from the Tap owners on this question.

6

Thank you very much!

7

Reviewed this with the TAP feed owners, and there’s no timed buffering happening on the Couchbase Server side - requests are sent to the feed immediately. Based on some other testing, this may have something to do with the way sync gateway is doing writes. If I run the same test with Sync Gateway listening to the Tap Feed, but another client doing the writes to the server, I don’t see the same 2-second groupings - the tap events show up as they are written.

Filed issue #616 against Sync Gateway for this issue

8

Thank you again! I’ll be following this issue then.

9

Has there been any updates on this issue? We are also interested in reducing the latency of the sync times.